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SURVEY  SOFTWARE  EVALUATION 


EXECUTIVE  SUMMARY 


Research  Requirement: 

More  than  ever,  computers  and  the  Internet  are  being  used  to  collect  survey  data  from  a 
variety  of  populations.  Web-based  surveys  have  many  advantages  (e.g.,  cost-effective  and 
efficient),  but  also  present  some  challenges  (e.g.,  ensuring  the  protection  of  personally 
identifiable  information).  Because  of  these  challenges,  the  United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  contracted  with  ICE  International  (ICE),  a  global 
research  and  consulting  firm,  to  conduct  needs  and  requirements  analyses  and  identify  the 
commercial  off  the  shelf  (COTS)  survey  software  programs  which  best  meet  ARFs  needs. 

Procedure: 

This  report  summarizes  findings  and  presents  recommendations  on  the  COTS  Web-based 
survey  software  products  which  meet  the  needs  of  ARI  survey  researchers,  can  be  housed  on 
Army  Knowledge  Online  (AKO)  or  ARFs  server,  and  are  compliant  with  Army  and  Department 
of  Defense  (DoD)  information  assurance  regulations \  ICE  first  conducted  a  needs  analysis  to 
identify  the  survey  needs  of  ARI  researchers.  To  determine  ARFs  survey  software  needs,  ICE 
conducted  two  focus  groups  with  ARI  staff,  five  individual  interviews  with  ARI  staff  and  an  ARI 
contractor,  and  disseminated  a  paper  and  pencil  survey  to  ARI  staff.  The  focus  groups,  individual 
interviews,  and  survey  enabled  ICE  to  ascertain  ARFs  survey  software  needs,  software  feature 
criticality,  and  information  technology  (IT)  requirements.  ICE  team  members  then  conducted  a 
market  analysis  of  74  COTS  Web-based  survey  software  products.  The  findings  and 
recommendations  within  this  report  are  also  based  on  a  comprehensive  review  of  information 
assurance  regulations,  Internet  searches  of  potential  survey  software  products,  telephone 
software  vendor  meetings,  and  software  testing  of  a  subset  of  products  that  most  closely  meet 
ARFs  research  needs  and  technical  requirements.  A  summary  of  the  major  findings  in  the  COTS 
survey  software  selection  is  presented  below. 

Findings: 

ICE  reviewed  74  COTS  Web-based  survey  software  products  for  potential  use  by  ARI 
researchers  in  survey  development  and  administration.  Products  were  reviewed  based  on  their 
ability  to  meet  the  following  key  criteria:  1)  survey  development  and  hosting  could  be  on  ARI 
servers;  2)  company  size  and  reputation  suggest  the  company  would  be  able  to  provide  quality 
service  for  many  years;  3)  the  survey  product  meets  Army  and  DoD  security  regulations  and 
technical  requirements,  and  4)  the  survey  products  meet  the  feature  and  function  needs  of  ARI 
researchers.  Based  on  our  review  and  analyses,  we  conclude  that  Confirmit  EFM  Professional 
and  Vovici  EFM  Community  are  the  two  COTS  Web-based  survey  software  products  that  best 
satisfy  ARFs  research  needs  and  technical  requirements.  Confirmit  EFM  Professional,  although 
expensive  compared  to  the  other  products  reviewed,  has  a  high  percentage  of  desired  survey 


*  The  following  Army  and  DoD  information  assurance  regulations  were  reviewed;  Army  Regulation  (AR)  25-1,  AR 
25-2,  Department  of  Defense  Instructions  (DoDI)  8500-2,  DoDI  8510-01,  DoDI  8580-1,  and  DoDI  8910-01 
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features  available  (81.58%)  and  is  compatible  with  the  IT  and  security  requirements  of  ARI. 
Vovici  EFM  Community  has  a  slightly  lower  percentage  of  survey  features  available  (72.37%) 
than  Confirmit  EFM  Professional,  but  is  a  more  cost-effective  solution  and  has  a  Certificate  of 
Networthiness  (CoN)  for  an  earlier  version  of  the  current  product. 

Utilization  and  Dissemination  of  Findings: 

With  the  increased  use  of  data  collections  via  Web-based  surveys  and  the  rapidly 
evolving  Web-based  survey  software  industry,  it  is  important  to  periodically  evaluate  COTS 
survey  software  programs  to  identify  the  product  that  best  fits  a  company’s  needs.  ARI  should 
consider  these  research  data  when  evaluating  COTS  Web-based  survey  software  products. 


^  Army  Networthiness  Certification  ensures  all  Automated  Information  Systems  (AIS)  on  the  Army  Network  are 
certified  as  to  the  capabilities,  limitations,  and  potential  impact  to  the  Army  Knowledge  Enterprise  (AKE).  It  also 
allows  the  Army’s  Chief  Information  Officer  (CIO)/G-6  and  U.S.  Army  Network  Enterprise  Technology  Command 
(NETCOM)  to  establish  accountability  of  and  manage  change  to  the  Army  Network. 
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INTRODUCTION 


More  than  ever,  computers  and  the  Internet  are  being  used  to  collect  survey  data  from  a 
variety  of  populations,  including  large-scale  surveys  of  the  active  duty  military  and  reserve 
populations  (e.g.,  by  the  United  States  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  [ARI]  or  the  Defense  Manpower  Data  Center  [DMDC]).  Web-based  surveys  have 
many  advantages:  they  are  cost-effective  (e.g.,  eliminating  postage),  efficient  (e.g.,  requiring 
little  database  cleaning  following  survey  fielding),  and  effective  (e.g.,  allowing  for  use  of  built-in 
skip  patterns).  Despite  these  advantages,  the  primary  challenges  of  Web-based  surveys  are:  1) 
identifying  the  best  Commercial  Off  the  Shelf  (COTS)  Web-based  survey  packages  to  serve  the 
particular  agency,  and  2)  maintaining  the  level  of  Web-based  security  to  ensure  data  safety  and 
sensitivity.  Based  on  these  challenges,  ARI  contracted  with  ICF  International  (ICF),  a  global 
research  and  consulting  firm,  to  conduct  needs  and  requirements  analyses  and  to  evaluate  COTS 
survey  software  programs. 

The  purpose  of  this  project  was  to  identify  COTS  Web-based  survey  software  products 
that  meet  the  needs  of  ARI  survey  researchers,  can  be  housed  on  Army  Knowledge  Online 
(AKO)  or  ARI’s  server,  and  are  compliant  with  Army  and  Department  of  Defense  (DoD) 
regulations  and  policies  on  information  assurance  and  other  relevant  guidance.  Under  this 
delivery  order,  ICF  performed  the  following  tasks  to  identify  commercial  survey  software 
packages  that  best  meet  ARI’s  survey  research  needs  while  adhering  to  Army  and  DoD 
information  assurance  guidance: 

1.  Conduct  project  administration  tasks  (e.g.,  kick-off  meeting,  project  plan,  In-Process 
Reviews  [IPR]) 

2.  Perform  an  analysis  of  ARFs  survey  software  needs 

3.  Review  information  assurance  regulations 

4.  Perform  a  market  analysis  of  COTS  survey  software  products 

5.  Perform  a  cost  analysis. 

This  report  summarizes  findings  and  presents  recommendations  regarding  the  COTS 
survey  software  products  that  best  meet  ARFs  survey  research  needs  and  are  consistent  with 
Army  and  DoD  guidance.  The  report  findings  and  recommendations  are  based  on  input  gathered 
from  focus  groups  with  twelve  staff  members  at  ARI  headquarters  in  Arlington,  VA.,  interviews 
with  five  ARI  staff  members  and  an  ARI  contractor,  numerous  telephone  meetings  with  survey 
vendor  representatives,  and  software  testing  and  evaluation  of  the  final  four  COTS  survey 
product  candidates  to  assess  the  presence  and  the  usability  of  survey  features. 

The  sections  of  this  report  include: 

■  Needs  Analysis — In  this  section,  we  present  our  methodology  and  results  regarding 

the  identification  of  ARFs  survey  software  requirements. 

■  Information  Assurance  Regulation  Review — In  this  section  we  present  our 

methodology  and  results  regarding  our  review  of  Army  and  DoD  information 
assurance  guidance  related  to  online  data  collection  and  survey  software. 
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■  Market  Analysis — In  this  section  we  present  our  methodology  and  results  related  to 

the  search  for  the  COTS  Web-based  survey  software  products  that  best  meet 
ARI’s  research  needs. 

■  Cost  Analysis — In  this  section  we  present  our  methodology  and  results  related  to 

pricing  of  the  four  finalist  survey  software  products. 

■  Conclusions  and  Recommendations — In  this  section  we  present  our  conclusions  and 

recommendations . 


Throughout  this  report  “developer”  refers  to  the  individual  who  programs  a  questionnaire 
(i.e.,  inputs  survey  questions  and  formatting)  into  Web-based  survey  software.  “Participant” 
refers  to  the  individual  answering  a  survey  questionnaire. 

Each  of  the  above  sections  is  presented  below. 
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NEEDS  ANAEYSIS 


ICE  conducted  an  analysis  of  ARI’s  Web-based  survey  software  requirements.  Data 
gathered  from  focus  groups  and  interviews  with  ARI  staff  members  were  used  to  develop  criteria 
for  the  utilities  and  functionalities  of  software  applications.  The  methodology  and  results  of  the 
needs  analysis  are  summarized  below. 


Methodology 

ICE  performed  the  following  five  subtasks  to  conduct  the  needs  analysis: 

1 .  Solicit  lists  of  desired  functions  and  features  from  ARI  staff, 

2.  Develop  focus  group  protocol, 

3.  Conduct  focus  groups  and  distribute  criticality  questionnaire, 

4.  Analyze  focus  group  data,  and 

5.  Develop  list  of  criteria  for  market  analysis. 

The  methodology  of  each  needs  analysis  subtask  is  detailed  below. 

Solicit  Eists  of  Desired  Eunctions  and  Eeatures  from  ARI  Staff 

ICE  software  evaluation  team  members  developed  categories  of  Web-based  survey 
software  features  (see  Appendix  A).  This  list  of  functions  and  features  was  provided  to  ARI 
researchers  in  developing  their  own  lists  of  desired  functions  and  features  of  COTS  Web-based 
survey  software  platforms. 


Develop  Eocus  Group  Protocol 

ICE  team  members  analyzed  ARI  staffs  desired  functions  and  features  and  developed  the 
focus  group  protocol.  The  focus  group  protocol  contained  questions  regarding  the  major 
strengths  and  weaknesses  of  currently  used  Web-based  survey  software  in  uploading, 
distributing,  and  downloading  surveys,  other  survey  needs  of  ARI  researchers,  and  the  criticality 
of  desired  functions  and  features  for  newly  purchased  survey  software.  A  copy  of  the  focus 
group  protocol  appears  in  Appendix  B. 

Conduct  Eocus  Groups  and  Distribute  Criticality  Questionnaire 

Using  the  focus  group  protocol,  evaluation  team  members  conducted  two  focus  groups 
(five  individuals  attended  the  first  session  and  seven  individuals  attended  the  second  session)  at 
ARI  headquarters  in  Arlington,  VA.  Eocus  groups  contained  the  appropriate  personnel  from 
ARI’s  Army  Trends  Analysis  Research  Unit  (ATARU),  Army  Personnel  Survey  Office  (APSO), 
Occupational  Analysis  Office  (OAO),  and  Personnel  Assessment  Research  Unit  (PARU).  A 
short  paper-based  survey  questionnaire  was  completed  by  focus  group  participants  to  aid  in  the 
assessment  of  feature  criticality.  This  criticality  questionnaire  appears  in  Appendix  C. 
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Analyze  Focus  Group  Data 


ICF  used  qualitative  data  analysis  techniques  to  analyze  the  focus  group  data,  identifying 
themes  related  to  functionality  requirements  and  the  criticality  of  each  requirement.  The  themes 
that  emerged  from  the  focus  groups  are  the  basis  for  the  market  analysis  criteria  found  in  Table 
1. 


Develop  List  of  Criteria  for  Market  Analysis 

ICF  compiled  a  list  of  ARI  staffs’  critical  and  non-critical  features  and  functions  for 
COTS  survey  software,  based  on  information  from  the  focus  groups,  individual  interviews,  and 
criticality  questionnaire.  These  features  and  functions  became  the  criteria  used  to  evaluate  COTS 
survey  software  products  in  the  market  analysis. 

Results 


The  analyses  of  information  from  the  focus  groups,  individual  interviews,  and 
questionnaires  resulted  in  a  list  of  78  survey  features/functions  and  their  criticality,  reflecting  the 
current  needs  of  ARI  researchers  (see  Table  1).  The  criteria  are  separated  into  four  levels  of 
criticality:  1)  required,  2)  critical  to  survey  functionality,  3)  improves  survey  functionality 
greatly,  and  4)  improves  survey  functionality  somewhat.  These  criteria  levels  determined  the 
order  in  which  product  features  were  reviewed. 


Table  1 

Market  Analysis  Criteria 
Item 
Number 


Criteria 


REQUIRED _ 

1 . 1  *Software  to  develop  surveys  can  be  installed  on  ARI  servers 

1.2  *Software  to  host  surveys  and  survey  database  can  be  installed  on  ARI  servers 

1.3  *Developer  interface  is  considered  "easy-to-use"  by  individuals  who  are  not  proficient  in 
Web  development 

1.4  *The  software  vendor  must  be  recognized  as  a  quality  vendor  and  is  expected  to  be 

_ offering/supporting  the  software  for  3  years  after  purchase _ 

CRITICAL  TO  SURVEY  FUNCTIONALITY _ 

2.01  *Meets  Army  and  DoD  security  regulations 

2.02  *Ability  to  export  data  to  a  format  compatible  with  SPSS  or  SAS 

2.03  *Ability  to  do  advanced  branching  forward  in  the  survey 

2.04  *  Ability  to  filter  reminder  email  recipients  (e.g.,  send  to  specific  sub-samples  based  on 

rank) 

2.05  Unlimited  number  of  responses  to  surveys 
2.06  Thorough  and  clear  documentation/training 


^  Note  that  there  are  no  criteria  with  the  following  numbers:  3.19,  3.23,  3.34,  3.36,  and  4.02.  In  the  initial  criteria  list 
developed  (reported  in  the  first  In-Process  Review  [IPR]),  these  criteria  were  mistakenly  included,  but  were 
duplicates  of  criteria  2.13,  2.25,  3.06,  2.10,  and  2.21,  respectively.  The  criteria  numbers  in  Table  1  were  kept  the 
same  as  were  listed  in  the  first  IPR  for  consistency. 
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Criteria 


Item 
Number 

2.07  Ability  to  create  professional-looking  surveys 

2.08  User-friendly  developer  interface,  including  ease  of  formatting  question  and  response 
option  text  (e.g.,  bold,  italics) 

2.09  Ability  to  skip/ask  particular  questions  based  upon  record  data/demographic  data  (i.e. 
piping  data  from  preloaded  demographic  data  file) 

2.10  Ability  to  show  previous  answers  in  the  text  of  current  questions  (i.e.  piping  into 
question  text) 

2.11  Ability  to  add  HTML  code  to  question  text  and/or  question  responses 

2.12  Advanced  formatting  (e.g.,  line  spacing,  line  indent,  column  labels,  line  breaks,  response 
column  widths,  wrapping  in  columns) 

2.13  Ability  to  write/code  survey  branching  logic  (rather  than  only  having  the  option  to  use  a 
drop  down  list) 

2.14  Ability  to  name  (label)  questions 

2.15  Ability  to  add  comment  boxes,  so  participants  can  add  comments  throughout  the  survey 

2.16  Ability  to  add  graphics  in  the  header  (at  beginning  of  survey) 

2.17  Ability  to  add  graphics  in  the  question  text 

2.18  Ability  to  edit  labels  of  navigation  buttons  (e.g.,  help  button,  frequently  asked  questions 

[FAQ]  button,  privacy  act  button,  submit  button) 

2.19  Customizable  templates  (e.g.,  theme  or  cascading  style  sheets) 

2.20  Spell  Check 

2.21  Offer  respondents  an  easy  way  to  review  their  answers  to  previous  questions  (i.e. 
without  having  to  backtrack  through  every  question)  (e.g.,  jump  function) 

2.22  Ability  to  print  out  the  question  properties  for  the  survey  (e.g.,  page,  question  variable 
name,  question  text,  and  question  scale) 

2.23  Ability  to  export  survey  question  text  and  response  options  to  external  document  (e.g., 
Microsoft  [MS]  Word) 

2.24  *Ability  to  remove  a  respondent  from  the  survey  distribution  list  during  the  fielding 
period  so  they  do  not  receive  future  correspondence  (e.g.,  reminder  email) 

2.25  Ability  to  track  the  exact  question  the  participant  answered  prior  to  abandoning  the 
survey 

2.26  Ability  to  automatically  create  and  assign  IDs/passwords 

2.27  Ability  to  filter  email  recipients  (e.g.,  send  only  to  participants  who  have  not  completed 
the  survey,  send  initial  email  based  on  a  participant  database  variable  to  overcome  limits 
on  sending  mass  emails) 

2.28  Ability  to  export  variable  names  and  labels  (with  data)  for  use  in  SPSS  or  SAS 

2.29  Ability  to  assign  variable  labels  within  the  application 

2.30  Ability  to  assign  export  response  option  values 

2.31  Ability  for  participant  to  withdraw  a  survey  (either  one  that  has  been  electronically 
submitted  or  one  that  is  partially  complete) 

IMPROVES  SURVEY  FUNCTIONALITY  GREATLY 

3.01  Ability  to  assign  export  variable  names 

3.02  Ability  to  export  data  directly  to  SAS 
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Item 

Number 


Criteria 


3.03  Ability  to  have  different  types  of  response  options  in  the  same  matrix  question 

3.04  Ability  to  Automatic  number  with  the  option  of  displaying  numbers  or  not 

3.05  Ability  to  add  to  and  select  from  a  response  option  library  of  commonly-used  response 

option  sets 

3.06  Ability  to  add  to  and  select  from  a  question  library  of  full  questions  (question  text  and 
response  options) 

3.07  Ability  to  show  only  the  response  options  that  were  selected  (or  not)  in  a  previous 
question  (i.e.  piping  into  response  options) 

3.08  Ability  to  randomize  the  order  of  response  options 

3.09  Automatically  add  response  values  as  each  response  is  added  (e.g.,  summing  percents) 

3.10  Ability  for  system  to  keep  historical  log  files  (i.e.,  that  record  upload,  deletion,  and 
emails  to  survey  respondents)  that  cannot  be  deleted  by  ARI 

3.11  Ability  to  export  data  directly  to  Microsoft  Access 

3.12  Ability  to  randomize  the  order  of  questions 

3.13  Ability  to  randomize  the  order  of  pages 

3.14  Presence  of  qualitative  data  analysis  functionality  (e.g.,  keying  in  on  words  or  phases) 

3.15  Ability  to  supplement  easy  developer  interface  with  HTML  or  other  coding  for  more 
control 

3.16  Ability  to  create  online  help  features  (e.g.,  pop-up  boxes  or  a  mouse  roll-over  function) 
that  displays  definitions 

3.17  Ability  to  import  survey  questions  from  external  application,  such  as  MS  Word 

3.18  Ability  to  branch  participants  backwards  in  the  survey 

3.20  Ability  to  export  identical  survey  (i.e.,  formatting  included)  to  external  document  (e.g., 
MS  Word) 

3.21  Ability  to  add  comments  to  the  survey  once  it  is  exported 

3.22  Ability  to  change  survey  questions  or  response  options  after  survey  is  in  field  without 
corrupting  the  data 

3.24  Ability  to  do  ranking  and  rating  questions  in  one  question  type 

3.25  Ability  for  respondents  to  withdraw  a  survey  (e.g.,  comparable  to  filling  out  a  paper 

form  and  decide  not  to  submit  it) 

3.26  Ability  to  do  sampling  for  extremely  long  surveys,  so  each  participant  does  not  have  to 
complete  entire  survey 

3.27  Ability  to  have  different  response  options  within  one  matrix  question  (e.g.,  sometimes 
have  a  Not  Applicable  (N/A)  response  option  and  other  times  not  having  N/A) 

3.28  Ability  to  attach  external  documents  to  the  dissemination  email  (e.g.,  a  support  letter 
from  an  Army  official)  or  be  able  to  imbed  email  signatures  into  the  dissemination  email 

3.29  Ability  for  participant  to  clear  a  page  or  a  specific  question  (e.g.,  radio  button) 

3.30  Ability  to  export  data  directly  to  SPSS  or  SAS 

3.31  Ability  for  system  to  automatically  create  an  archive/back-up  of  questionnaire/data 

3.32  Ability  to  perform  Advanced  Page  Layout  (e.g.,  resize  objects,  layer  objects,  snap 
objects  to  a  grid,  and  lock  a  page) 

3.33  Ability  to  name  (label)  pages 
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Criteria 


Item 
Number 

3.35  Ability  to  have  administrative  control,  such  as  being  able  to  remove  cases  or  reset  cases 
(rather  than  going  into  U.S.  Army  Information  Management  Support  Center  [IMCEN]  to 
do  so)  to  allow  participants  who  were  inappropriately  exited  from  the  survey  back  in 
IMPROVES  SURVEY  FUNCTIONALITY  SOMEWHAT 

4.01  Ability  to  add  graphics  in  the  response  options 

4.03  Presence  of  an  option  to  export  a  subset  of  questions  vs.  entire  survey 

4.04  Presence  of  optical  scanning  features 

4.05  Ability  for  platform  to  automatically  dump  test  data  once  activated 

4.06  Presence  of  a  feature  that  allows  developers  to  easily  check  if  skip  patterns  work  without 
taking  entire  survey  as  participant 

4.07  Ability  to  do  data  cleaning  and  some  data  analysis  in  the  application 

4.08  Ability  to  display  basic  figures/tables  (e.g.,  cross  tab)  of  certain  important  questions 

(e.g.,  ability  to  look  at  non-responses  by  rank) 

4.09  Presence  of  a  spell  check  library  where  commonly-used  words  or  acronyms  can  be 

4.10  Presence  of  meta-data  capabilities 

4.11  Ability  to  store  the  data  in  an  encrypted  format 

4.12  Ability  to  create  professional-looking  emails 

4.13  Ability  to  publish  a  survey  to  a  transportable  file  that  can  be  run  on  local  machines  (e.g., 
non-Internet) 

*  Denotes  required  features 

Note.  Criteria  3.19,  3.23,  3.34,  3.36,  and  4.02  were  eliminated  because  of  duplication  with  other  criteria  listed. 
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INFORMATION  ASSURANCE  REGULATION  REVIEW 


The  methodology  and  results  of  the  review  of  information  assurance  regulations  relevant 
to  online  data  collections  are  summarized  below. 

Methodology 

To  review  Army  and  DOD  information  assurance  regulations,  ICE  compiled  and 
examined  the  relevant  regulations  and  developed  a  list  of  the  relevant  portions  of  each 
regulation. 

ARI  provided  ICE  with  the  appropriate  Army  and  DoD  information  assurance 
regulations,  including  the  following: 

■  Army  Regulation  (AR)  25-1:  Army  Knowledge  Management  and  Information 
Technology 

■  Army  Regulation  (AR)  25-2:  Information  Assurance 

■  Department  of  Defense  Instructions  (DoDI)  8500-2:  Information  Assurance 
Implementation 

■  Department  of  Defense  Instructions  (DoDI)  8510-01:  Information  Assurance 
Certification  and  Accreditation  Process 

■  Department  of  Defense  Instructions  (DoDI)  8580-1:  Information  Assurance  in  the 
Defense  Acquisition  System 

■  Department  of  Defense  Instructions  (DoDI)  8910-01:  Information  Collection  and 
Reporting. 

The  ICE  project  team  then  worked  with  ICE  information  technology  (IT)  and  security 
personnel  to  review  the  appropriate  Army  and  DoD  regulation  documents  and  extract  and 
simplify  the  regulations  that  are  relevant  to  online  data  collection  and  Web-based  survey 
software. 

Einally,  ICE  compiled  a  list  of  regulations  that  are  relevant  to  online  data  collection 
(found  in  Appendix  D).  This  list  was  used  as  the  basis  of  the  market  analysis  and  security  review 
task. 


Results 

Table  2  presents  a  summary  of  Army  and  DoD  regulations  that  are  relevant  to  online  data 
collection  and  storage  of  information  on  Army  servers.  Although  other  regulations  mention  data 
collection,  AR  25-1  and  AR  25-2  cover  the  subject  in  depth  and  are  most  relevant.  Many  of  these 
regulations  authorize  the  local  information  assurance  (lA)  organization  (i.e..  United  States  Army 
Information  Management  Support  Center  [IMCEN])  to  make  lA  decisions.  IMCEN  is  still 
developing  many  of  its  regulations,  so  until  those  are  published,  the  COTS  product  chosen  is 
subject  to  evolving  standards.  A  more  detailed  breakdown  of  the  regulations  summary,  including 
the  regulation  title  and  language,  appears  in  Appendix  D. 
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Table  2 


Summary  of  Relevant  Army  and  DoD  Regulations 


mn 

Summary  of  Relevant  Text 

AR  25-1 

5-3 

The  highest  security  level  of  ARI  survey  data  is  confidential  and  so  is  not 
subject  to  regulation  beyond  AR  25-1,  AR  25-2,  and  DoDI  5200.40. 

AR  25-1 

5-5 

The  U.S.  Army  Information  Management  Support  Center  (IMCEN)  must 
approve  all  software  installed  on  its  servers. 

AR  25-1 

5-5 

The  highest  security  level  of  ARI  survey  data  is  confidential,  and  so  the 
database  management  systems  (DBMS)  must  be  protected  according  to 
IMCEN  standards  for  this  type  of  data. 

AR  25-1 

6-1  j. 

All  products  of  the  software  (i.e.  surveys)  are  the  property  of  the  U.S. 
Government.  Eicense  agreement  cannot  include  any  ownership  of  the 
products  by  the  software  vendor. 

AR  25-1 

6-1  r. 

The  software  chosen  does  not  need  to  be  supported  by  IMCEN  (in  terms 
of  training  and  troubleshooting). 

AR  25-1 

6-2  g. 

Preference  is  to  be  given  to  software  that  will  be  supported  and  continue 
to  be  upgraded  (i.e.,  preference  should  be  given  to  a  vendor  that  has  a 
strong  likelihood  of  existing  throughout  the  lifecycle  of  ARFs  survey 
software  need). 

AR  25-1 

6-2  g. 

Data  should  be  stored  in  a  format  independent  of  software  vendor. 

AR  25-1 

6-2  g. 

Data  should  be  accessible  in  a  standard  format  (i.e.  one  approved  by 
IMCEN). 

AR  25-1 

6-2  e.  (3) 

Preference  should  be  given  to  COTS  already  in  DoD  inventory. 

AR  25-1 

6-2  e.  (3) 

Purchases  must  be  coordinated  with  the  Army  Small  Computer  Program 
Office  (ASCPO). 

AR  25-1 

6-2  m. 

Preference  should  be  given  to  COTS  products  that  include  upgrades  for  at 
least  3  years  after  purchase. 

AR  25-1 

6-4  n. 

Authentication  must  be  achieved  through  Army  Knowledge  Online 
(AKO)  Eightweight  Directory  Access  Protocol  (EDAP)  or  a  waiver  must 
be  granted. 

AR  25-1 

6-4  n. 

Secure  Sockets  Eayer  (SSE)  must  be  enabled  using  DoD  public  key 
infrastructure  (PKI)  certificates  for  server  authentication  and  client/server 
authentication. 

AR  25-1 

6-4  n. 

The  minimum  security  provided  for  sensitive  by  unclassified  information 
is  SSE. 

AR  25-2 

4-5  c. 

An  audit  log  must  be  available  for  the  development  of  the  survey  and  for 
the  fielding  of  the  survey. 

AR  25-2 

4-5  c. 

Authentication  of  developers  must  include  a  unique  identifier. 

AR  25-2 

4-5  c. 

Authentication  of  developers  should  at  a  minimum  be  a  two-factor 
mechanism. 

AR  25-2 

4-5  c. 

The  COTS  product  should  be  configurable  to  not  allow  anonymous 

AR  25-2 

4-5  c. 

The  COTS  product  should  provide  an  auditable  login  log  for  developers. 

AR  25-2 

4-5  c. 

The  COTS  product  should  be  configurable  to  prevent  logins  after  rules  set 
forth  in  a  pre-determined  protocol  have  been  met. 

AR  25-2 

4-5  f. 

Any  installation  of  a  COTS  product  must  be  coordinated  with  IMCEN. 
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■ASH 

Summary  of  Relevant  Text 

AR  25-2 

4-6 

License  agreement  must  be  reviewed  to  ensure  it  covers  all  uses  needed 
by  ARI. 

AR  25-2 

4-7 

The  COTS  product  will  not  use  its  own  server  application. 

AR  25-2 

4-7 

The  database  must  reside  on  a  trusted  server. 

AR  25-2 

4-20  e. 

(7) 

The  COTS  product  will  use  only  approved  access  controls  between  the 
participant  interface  and  the  survey  database. 
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MARKET  ANALYSIS 


ICE  conducted  a  market  analysis  in  which  available  COTS  survey  software  products  were 
evaluated  against  the  criteria  developed  in  the  needs  analysis.  The  methodology  and  results  of 
the  market  analysis  are  summarized  below. 

Methodology 

The  methodology  employed  for  the  market  analysis  included  Internet  searches,  software 
vendor  telephone  meetings,  and  software  testing  and  evaluation.  This  analysis  included  two 
major  sub  tasks: 

1.  Identify  and  review  COTS  Web-based  survey  software,  and 

2.  Test  candidate  software. 

The  methodology  of  each  market  analysis  subtask  is  detailed  below. 

Identify  and  review  COTS  Web-based  survey  software 


ICE  performed  a  thorough  review  of  the  currently  available  COTS  survey  software 
products  based  on  AREs  needs  and  on  Army  and  DoD  information  assurance  guidelines. 

Develop  Initial  Candidate  Software  List.  To  develop  the  initial  list  of  software  candidates, 
evaluation  team  members  identified  software  packages  advertised  or  reviewed  in  mainstream 
publications,  such  as  computer  software  reviewers  (i.e.,  C-NET  and  PC  Magazine)  and  in 
common  search  engines  (i.e.,  Google  and  Yahoo)  as  of  October  2008.  This  search  resulted  in  an 
exhaustive  list  of  74  COTS  survey  software  products  to  review  (found  in  Appendix  E). 

Revise  List  Based  on  “Required”  Criteria  (Eirst  Round  of  Review).  Through  Internet  searches 
and  telephone  meetings  with  COTS  survey  software  vendors,  team  members  evaluated  the  initial 
list  of  74  COTS  survey  software  based  on  the  “required”  criteria.  These  “required”  criteria 
include:  1)  software  to  develop  surveys  can  be  installed  on  ARI  servers;  2)  software  to  host 
surveys  and  the  survey  database  can  be  installed  on  ARI  servers;  3)  the  developer  interface  is 
considered  "easy-to-use"  by  individuals  who  are  not  proficient  in  Web  development;  and  4)  the 
software  vendor  must  be  recognized  as  a  quality  vendor  and  be  expected  to  offer/support  the 
software  for  at  least  three  years  after  purchase.  The  4th  criterion  was  rated  on  a  5-point  Likert- 
type  scale  (1  =  solid,  5  =  poor),  based  on  the  following  components  and  processes: 

■  Reputation:  ICE  review  of  the  vendor  website,  discussions  with  colleagues  who  have 
similar  needs,  stock  history  if  publicly  traded,  or  other  financial  news,  if  available, 

■  Size:  Number  and  size  of  clients  listed  on  website, 

■  Quality:  Website  design  and  maintenance  (this  was  used  only  when  the  website  appeared 
to  be  of  amateur  quality),  and 

■  U.S.  Owned:  Determined  from  website. 

Products  advanced  to  the  second  round  of  review  if  the  vendor  reported  that:  1)  their  survey 
development  software  could  be  installed  on  ARI  servers;  2)  the  hosting  software  and  software 
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databases  could  be  installed  on  ARI  servers;  3)  the  hosting  software  allowed  developers  to  enter 
survey  questions  and  manipulate  survey  format  through  a  graphical  user  interface  (GUI)  (i.e., 
non-coding  interface);  4)  the  company’s  reputation,  size,  and  quality  combined  was  rated  1-4 
(i.e.,  not  rated  5  =  poor);  and  5)  the  company  was  headquartered  within  the  United  States. 

Revise  List  Based  on  “Critical”  Criteria  (Second  Round  of  Review).  IGF  evaluation  team 
members  reviewed  the  survey  software  programs  that  were  advanced  to  the  second  round  of 
review  based  on  the  “critical”  criteria  identified  in  the  needs  analysis.  A  few  examples  of  the 
“critical”  criteria  used  for  this  round  of  review  include  the:  1)  ability  to  export  data  to  a  format 
compatible  with  SPSS  or  SAS;  2)  ability  to  do  advanced  branching  forward  in  the  survey;  3) 
ability  to  filter  reminder  email  recipients  (e.g.,  send  to  specific  sub-samples  based  on  rank);  and 
4)  ability  to  remove  a  respondent  from  the  survey  distribution  list  (during  the  fielding  period)  so 
they  do  not  receive  future  correspondence  (e.g.,  reminder  email).  A  full  list  of  the  “critical” 
criteria  used  in  this  round  of  review  can  be  found  in  Table  1.  Products  advanced  to  the 
subsequent  third  round  of  review  if  the  vendor  reported  that  all  of  the  required  criteria  and  more 
than  85%  of  the  “critical”  criteria  and  were  available.  Two  vendors— Business  Objects,  an  SAP 
Company  and  iMagic  Software— were  unresponsive  when  contacted.  These  two  vendors  were 
eliminated  because  their  lack  of  responsiveness  was  potentially  indicative  of  the  low  level  of 
support  a  new  customer  might  receive. 

Revise  List  Based  on  “Improves  Greatly”  Criteria  (Third  Round  of  Review).  Evaluation  team 
members  reviewed  the  survey  software  programs  that  were  advanced  to  the  third  round  of  review 
based  on  the  “improves  greatly”  criteria  identified  in  the  needs  analysis.  A  few  examples  of  the 
“improves  greatly”  criteria  used  for  this  round  of  review  include  the:  1)  ability  to  assign  export 
variable  names;  2)  ability  to  export  data  directly  to  SAS;  3)  ability  to  randomize  the  order  of 
questions;  and  4)  ability  to  do  ranking  and  rating  questions  in  one  question  type.  A  full  list  of  the 
“improves  greatly”  criteria  used  in  this  round  of  review  can  be  found  in  Table  1.  Products 
advanced  to  the  subsequent  software  testing  round  if  the  vendor  reported  that  greater  than  90% 
of  the  “improves  greatly”  criteria  were  available. 

Software  Testing  (Fourth  Round  of  Review).  Four  products  advanced  to  the  fourth  and  final 
round  of  review  and  were  subject  to  software  testing  and  evaluation.  IGF  coordinated  with  the 
four  final  candidate  survey  software  vendors  to  obtain  access  to  the  software  needing  for  testing 
and  evaluation. 

Preparation  for  testing  of  finalist  products.  Prior  to  testing  the  four  finalist  products, 
evaluation  team  members  met  with  an  IMCEN  representative  to  identify  the  typical  IMCEN 
testing  procedures  and  software  certification  process  and  to  identify  ARFs  local  IT  and  security 
requirements.  This  information  was  used  to  help  develop  the  IT  and  security  requirements  and 
software  testing  protocol.  IGF  team  members  also  conducted  telephone  meetings  with  and 
solicited  information  from  all  finalist  survey  software  product  vendors  to  assess  whether  the 
survey  software  product  would  meet  ARFs  IT  and  security  needs/requirements.  A  few 
examples  of  the  IT  and  security  information  gathered  from  IMCEN  and  software  vendors 
include:  1)  server  CPU/hardware  capacity;  2)  Web  server  supported;  and  3)  event  log  capability. 
A  full  list  of  IT  and  security  review  criteria  are  included  in  Table  4. 
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Based  on  the  market  analysis  criteria,  information  procured  from  IMCEN,  and  our  review 
of  Army  and  DoD  information  assurance  regulations,  ICF  developed  a  testing  protocol  and 
sample  questionnaire  for  ICF  staff  to  use  when  testing  the  4  finalist  COTS  survey  software 
products.  The  software  testing  protocol  appears  in  Appendix  F  and  the  sample  questionnaire 
appears  in  Appendix  G. 

Testing  of  finalist  products.  ICF  contacted  the  four  finalist  COTS  survey  software 
products  companies  to  obtain  downloadable  test  versions.  Current  market  practices  involve  trials 
that  use  software  on  the  application  developer’s  own  servers,  so  ICF  team  members  tested  the 
software  products  on  the  vendor’s  (i.e.,  application  developer)  servers.  Each  of  the  four  finalist 
vendors  indicated  that  the  candidate  software  products  tested  were  identical  in  features  and 
functionality  to  the  downloadable  versions  of  the  survey  software. 

Using  the  testing  protocol  and  sample  questionnaire,  evaluation  team  members  tested 
each  of  the  four  finalist  COTS  survey  software  products.  Two  ICF  team  members  independently 
tested  and  rated  each  feature’s  usability  on  a  5-point  Fikert-type  scale  from  1  (difficult)  to  5 
(easy).  Farge  discrepancies  in  ratings  (i.e.,  differences  of  2  or  more  points)  were  rectified  by  a 
consensus  group  decision. 

Collect  additional  company  and  support  information  for  finalist  products.  ICF  team 
members  collected  company  and  support  information  from  all  finalist  survey  software  vendors  to 
ensure  that  each  finalist  product  company  has  the  size  and  health  suitable  to  a  long-standing 
company.  The  company  and  support  information  collected  include:  1)  company  revenue  and 
profitability,  2)  total  number  of  employees,  3)  number  of  help  desk/support  employees,  and  4) 
number  of  research  and  development  employees. 

Results 

Three  major  aspects  of  the  survey  software  products  were  reviewed  in  the  market 
analysis:  1)  functions  and  features,  2)  IT  and  security,  and  3)  company  and  support.  The  results 
of  each  of  these  reviews  are  presented  below. 

Functions  and  Features  Review 


Of  the  74  initial  COTS  survey  software  products,  52  products  were  eliminated  in  the  first 
round  of  review  because  of  their  software  structure  (i.e.,  no  graphical  user  interface  [GUI]  to 
enter  survey  questions  and  formatting),  requirement  to  be  downloaded  on  survey  vendor’s 
servers,  or  inadequate  company  size,  reputation,  or  location  of  ownership  (i.e.,  companies  that 
were  not  United  States-based).  Of  the  remaining  22  COTS  survey  software  products  evaluated  in 
the  second  round  of  review,  13  products  were  eliminated  because  of  a  large  percent  (i.e.,  15%  or 
more)  of  unavailable  “critical”  features.  Of  the  remaining  9  COTS  survey  software  products 
evaluated  in  the  third  round  of  review,  5  products  were  eliminated  because  of  a  large  percent 
(i.e.,  10%  or  more)  of  unavailable  features  that  would  greatly  improve  survey  development  and 
administration.  The  reasons  and  stage  at  which  each  product  was  eliminated  are  listed  in 
Appendix  H.  The  following  four  products  passed  the  first  three  rounds  of  survey  functions  and 
features  review  and  were  included  in  the  software  testing  and  evaluation  phase  of  the  market 
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analysis:  Confirmit  EFM  Professional,  Vovici  EFM  Community,  WorldApp  KeySurvey,  and 
Zarca  Interactive. 


Four  finalist  products 

Table  3  lists  the  percent  of  features  available  and  the  average  usability  ratings  for  each 
round  of  review  and  overall  for  each  of  the  four  finalist  software  products.  The  percent  of 
features  available  was  calculated  by  summing  the  number  of  available  features  (as  determined  by 
company  report  and/or  software  testing)  divided  by  the  potential  number  of  features  studied  in 
that  round.  Usability  ratings  were  averaged  for  tested  features  (i.e.,  features  that  were  unavailable 
or  not  tested  were  not  included  in  the  average  usability  ratings).  A  more  detailed  comparison  of 
the  survey  software  finalist  products  appears  in  Appendix  J. 

Results  of  the  features  and  functions  review  for  Zarca  Interactive  suggest  that,  although 
available  features  achieved  decent  usability  ratings  (4.28/5),  the  product  has  significantly  fewer 
survey  features  available  than  the  other  products  tested.  Because  of  the  relatively  large  percent  of 
unavailable  features,  we  conclude  that  Zarca  Interactive  is  not  among  those  products  which 
optimally  meet  ARFs  needs. 

Results  of  the  features  and  functions  review  for  Confirmit  EFM  Professional,  Vovici 
EFM  Community,  and  WorldApp  KeySurvey  are  generally  positive.  While  WorldApp 
KeySurvey  possesses  the  highest  percent  of  available  features,  it  has  the  least  favorable  usability 
ratings  of  the  three  remaining  products.  Vovici  EFM  Community  has  the  best  usability  ratings, 
but  the  lowest  number  of  available  features  of  the  three  remaining  products.  Confirmit  EFM 
Professional  appears  to  be  the  most  powerful  product  with  a  good  balance  of  available  features 
and  easy  usability. 

It  is  important  to  note  that  the  presence  or  absence  of  each  survey  feature  was  assessed 
via  vendor  report  in  the  first  three  rounds  of  review.  ICF  team  members  found  in  the  testing 
round  of  the  functions  and  features  review  that  many  vendors  indicated  a  greater  number  of 
features  available  than  were  actually  available.  Therefore,  the  testing  results  for  the  finalist 
vendors  (found  in  Appendix  J)  may  display  a  lower  percentage  of  features  available  than  the 
previous  round  cutoff. 
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Table  3 


COTS  Survey  Software  Product  Comparison 


Survey  Vendors 

Features 

Confirmit 

EFM 

Professional 

Vovici  EFM 
Community 

WorldApp 

KeySurvey 

Zarca 

Interactive 

“Required”  Features 

%  of  Features  Available 

90.32% 

83.87% 

93.55% 

74.19% 

Average  Usability  Rating 

4.60 

4.67 

3.85 

4.07 

“Improves  Greatly”  Features 

%  of  Features  Available 

71.88% 

62.50% 

71.88% 

50.00% 

Average  Usability  Rating 

4.31 

4.54 

4.22 

4.67 

“Improves  Somewhat”  Features 

%  of  Features  Available 

84.62% 

69.23% 

84.62% 

61.54% 

Average  Usability  Rating 

4.88 

4.50 

4.33 

3.25 

Total  %  of  Features 
Available 

81.58% 

72.37% 

82.89% 

61.84% 

TOTAE  Average 
Usability  Rating 

4.54 

4.64 

4.06 

4.28 

Note:  see  detailed  breakdown  in  Appendix  J. 

Note:  Usability  ratings  scale  is  1  =  difficult  to  5  =  easy. 


Products  of  Particular  Interest.  A  more  detailed  analysis  of  four  products  of  particular  interest  to 
ARI  (i.e.,  Prezza  Technologies  Check  Box  Survey,  Questionmark  Perception,  SPSS  Desktop 
Author,  and  SPSS  Dimensions)  appears  in  Appendix  I.  Results  of  the  features  and  function 
review  for  SPSS  Dimensions  suggest  that  although  the  product  allows  for  93.55%  of  the 
“critical”  features  and  93.75%  of  the  “improves  greatly”  features,  the  product  has  a  graphical 
user  interface  (GUI)  development  limitation  of  100  pieces  of  information  (i.e.,  surveys  with  more 
than  100  pieces  of  information,  including  questions  and  instructions,  can  only  be  developed 
through  coding).  Because  of  this  limitation,  ICF  team  members  did  not  pass  SPSS  Dimensions 
into  the  software  testing  round  of  the  market  analysis. 

IT  and  Security  Review 


Confirmit  EFM  Professional,  Vovici  EFM  Community,  and  Zarca  Interactive  all  pass  the 
IT  and  Security  review  due  to  their  ability  to  easily  integrate  with  the  existing  ARI  and  AKO 
systems  using  a  Microsoft  .Net  framework  and  Internet  Information  Services  (IIS)  Web  servers. 
These  three  products  also  had  features  similar  to  their  competitors,  such  as  the  ability  to  hash 
passwords  and  having  Eightweight  Directory  Access  Protocol  (EDAP)  connectivity  using 
Microsoft  Exchange.  On  the  other  hand,  WorldApp  Key  Survey,  although  a  robust  application, 
does  not  support  a  Microsoft  .Net  frame  work  and  requires  use  of  Java  Runtime  Environment 
(JRE)  and  has  not  been  tested  using  IIS.  Therefore,  the  server  requirements  of  WorldApp  Key 
Survey  do  not  suit  ARFs  existing  infrastructure.  Table  4  presents  the  IT  requirements  and 
specifications  of  each  of  the  four  finalist  COTS  survey  software  products. 
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Both  Confirmit  EFM  Professional  and  Vovici  EFM  Community  have  undergone  and 
passed  thorough  security  reviews."^  Both  products  are  secure;  thus,  we  recommend  both  Vovici 
EFM  Community  and  Confirmit  EFM  Professional  on  the  basis  of  security.  Vovici  EFM 
Community  version  2.2  has  achieved  a  Certificate  of  Networthiness  from  the  Army.  The 
certified  version  of  Vovici  EFM  Community  is  an  earlier  version  of  the  product  currently  offered 
(i.e.,  4.0),  but  no  major  changes  have  been  made  that  should  impair  the  product’s  ability  to 
maintain  its  Certificate  of  Networthiness.  Nevertheless,  an  IMCEN  review  of  IT  requirements 
and  security  is  still  recommended. 

Table  4 


Web-Based  Survey  Software  Vendor  Information  Technology  (IT)  Specifications 


Survey  Vendor  Capabilities 

Requirements 

Confirmit 

EFM 

Professional 

Vovici  EFM  WorldApp 

Community  KeySurvey 

Zarca 

Interactive 

Server  CPU/Hardware 
Capacity 

Intel  Pentium 
Core  2  or 
higher 

Intel  Pentium  ■  Intel  Pentium  Core 
Core  2  or  higher  2  or  server  side 

processor  for  better 
performance 

■  AMD  Athlon  -  64, 
Sempron  or  server 
side  processor  for 
better  performance 

■  Apple  -  G5  or  better 

■  SUN  -  UltraSparc 

III  or  better 

Intel  Pentium 
Core  2  or  higher 
(min  required  is 
Pentium  4) 

Hard  Drive 

No  information 
provided 

80GB  (Web  and  ■  60GB  (Web  and  DB 
DB  Server  each)  Server  each) 

40GB(Web 
Server)  &  100GB 
(DB  Server) 

System  Memory 
(RAM) 

2GB(Web 
Server)  -4GB 
(DB  Server) 

IGB -2GB  ■ 2GB  -  4GB 

2GB(Web  Server) 

-  4GB (DB 
Server) 

Confirmit  EFM  Professional’s  security  review  was  the  Foundstone  Security  Assessment  (copyright  ©  2007 
McAfee,  Inc.)  conducted  by  Foundstone  Professional  Services,  A  Division  of  McAfee,  and  Vovici  EFM 
Community’s  security  review  was  documented  in  the  Certificate  of  Networthiness  approved  by  the  U.  S.  Army. 
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Survey  Vendor  Capabilities 

Confirmit 

FFM 

Vovici  EFM 

WorldApp 

Zarca 

Requirements 

Professional 

Community 

KeySurvey 

Interactive 

Web  Server 

Internet 

Internet 

■  Internet  Information 

Internet 

Supported 

Information 
Services  (IIS) 

Information 
Services  (IIS) 

Services  (IIS)  ver. 

5.0  &  6.0  (not  tested 
by  vendodr) 

■  Apache  1.3,  2.0  & 

2.2 

■  Webstar  (Mac  OS 

X) 

■  SunOne 

Information 
Services  (IIS) 

Database  Server 

MS  SQL 

■  MS  SQE  Server 

■  Oracle  9i,  lOg 

MS  SQE 

Supported 

Server 

■  Oracle  9i,  lOg 

■  MySQE 

■  MS  SQE  Server 

Server 

Operating  System 

Windows 

■  Windows  2000 

■  Any  Compatible 

Windows  2003 

Supported 

2003  Server 

Server  (32  bit) 

■  Windows  2000 
Advanced  Server 
(32  bit) 

■  Windows  2003 
Server  (32  bit) 

with  Java  2  SDK 

1.5 

■  Tomcat  5.5  (tested 
by  Vendor) 

Server 

Framework 

Supported 

•Net  2.0 

.NET  2.0 

JRE 

(Supports  Java 
Servlet  Specification 
v.2.4  and  5  Java 
ServerPages  v.2.0 
Specification) 

.NET  2.0 

Survey  Taking 

Internet 

■  Mozilla  Eirefox 

■  Mozilla  Eirefox  ver. 

■  Mozilla  Eirefox 

Browser  Types 
Supported 

Explorer  6.0 

ver.  1 .0  or  higher 

■  Internet  Explorer 
ver.  5.0  or  higher 

■  Safari  ver.  2.0  or 
higher 

2.0.x  or  higher 

■  Internet  Explorer 
ver.  6.0  sp2  or 
higher 

■  Safari  ver.  2.0  or 
higher 

1.5  or  higher 

■  Internet  Explorer 

5.5  or  higher 

■  Safari  ver.  2.0  or 
higher 

Survey  Creation 

Internet 

■  Mozilla  Eirefox 

■  Mozilla  Eirefox  ver. 

■  Mozilla  Eirefox 

Browser  Types 
Supported 

Explorer  6.0 

ver.  1.5  or  higher 
■  Internet  Explorer 
ver.  6.0  sp2  or 
higher 

2.0.x  or  higher 

■  Internet  Explorer 
ver.  6.0  sp2  or 
higher 

■  Safari  2.0  or  higher 

2.0  or  higher 

■  Internet  Explorer 
6.0  or  higher 

■  Safari  2.0  or 
higher 
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Survey  Vendor  Capabilities 

Requirements 

Confirmit 

EFM 

Professional 

Vovici  EFM 
Community 

WorldApp 

KeySurvey 

Zarca 

Interactive 

Application  & 

Database  Client 

Hosting  Capability 

Web  Application/ 
Internet  Connectivity 
Capability 

Password  Encryption 
Capability 

(SHA-256 

Algorithm) 

(MD5  Algorithm) 

Secure  Socket  Layer 

(SSL)  Capability 

(128  bit) 

(128  bit) 

(128  bit) 

(128  bit) 

American  with 
Disabilities  Act 
(ADA)  508 

Compliance 

Event  Log  Capability 

Integration  & 
Application 
Programming  Interface 
(API)  Capability 

Lightweight 

Directory  Access 
Protocol  (LDAP) 

(add-on 

module) 

Ability  to  Send  out 
emails  to  users 

v'  Feature  available 


Company  and  Support  Review 

ICF  gathered  company  and  support  information  for  each  of  the  four  finalist  COTS  survey 
software  products.  Table  5  lists  information  regarding  company  revenue  and  profitability,  total 
number  of  employees,  number  of  help  desk  and  support  employees,  and  number  of  research  and 
development  employees  for  each  finalist  company.  This  table  suggests  that  all  companies  have 
the  size  and  health  suitable  to  a  long-standing  company. 
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Table  5 


Web-Based  Survey  Software  Vendor  Company  and  Support  Information 


Survey  Vendor 

Confirmit 

Vovici  Corp 

WorldApp 

Zarca 

Interactive,  Inc. 

Company  and  Support  Information 

Company 

The  last  three 

A  private 

A  private 

A  privately  held 

revenue  and 

years  revenue  has 

company  owned 

company  that  is 

company  that  has 

profitability 

ranged  from 

by  a  multi¬ 

cash  flow  positive 

experienced  30- 

$17.3-$34.2and 

billion  dollar 

and  has 

50%  growth  in 

the  last  three 

venture  capital 

experienced 

each  of  the  last 

years’ 

firm  called 

steady  growth 

three  years. 

profitability  has 

Austin  Ventures 

throughout  its  7- 

ranged  from 
$.915  to  $4.6. 

year  history. 

Total  number  of 
employees 

250 

90 

100 

65 

Number  of  Help 

Desk/Support 

employees 

34 

20 

13 

12 

Number  of 
Research  & 
Development 
employees 

120 

20 

17 

45 

Sources:  Vendor  self-reports 


All  four  finalist  companies  service  high  profile  and  security-conscious  clients,  however, 
Vovici  Corporation  has  the  client  base  most  similar  to  ARI.  Table  6  lists  example  clients  for  each 
finalist  company. 
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Table  6 


Client  Examples  for  Web-Based  Survey  Software  Vendors 


Confirmit 

Vovici  Corp 

WorldApp 

Zarca 

Interactive,  Inc. 

■  British  Council 

■  Dept  of  Defense 

■  Federal  Aviation 

■  The  Conference 

■  Abt  SRBI  Inc. 

■  Dept  of  Homeland  Security 

Administration 

Board 

■  IP  Morgan  Chase 

■  US  Air  Force  (e.g..  Hill  Air 

(FAA) 

■  Union  College 

■  CitiGroup 

Force  Base  [AFB], 

■  Federal  Bureau  of 

■  Verizon 

■  Barclay's 

Kirtland  AFB,  Fackland 

Investigation  (FBI) 

■  HSBC 

AFB) 

■  Marine  Corps 

■  Prudential 

■  US  Army  (e.g..  Fort  Fee, 

Operational  Test 

■  Tufts  Health  Plan 

Fort  Benning,  Fort 

and  Evaluation 

■  Google 

Jackson) 

Activity 

■  HP 

■  US  Army  Soldier  Support 

(MCOTEA) 

■  Accenture 

Institute 

■  United  States 

■  Microsoft 

■  US  Coast  Guard 

Geological  Survey 

■  Manulife  Financial 

■  US  Joint  Forces  Command 

■  US  Marines  (e.g..  Center 
for  Fessons  Feamed) 

■  US  Navy  (e.g..  Navy 
Commander  Operational 
Test  and  Evaluation  Force, 
Naval  War  College,  Naval 
Personnel  Development 
Command,  US  Navy 
Inspector) 

■  Space  and  Naval  Warfare 
Center  (SPAWAR)US 
Military  Entrance 

Processing  Command 

(USGS) 

Sources:  Vendor  self-reports 


Market  Analysis  Recommendations 


The  survey  functions  and  features  review  revealed  that  Zarca  Interactive  did  not  meet 
ARI’s  research  needs  as  well  as  the  other  finalist  products;  thus  we  conclude  that  Zarca 
Interactive  is  not  among  those  products  which  optimally  meet  ARI’s  needs.  The  market  analysis 
IT  and  security  review  revealed  that  WorldApp  KeySurvey  utilizes  a  framework  that  is 
incompatible  with  ARI’s  existing  infrastructure;  thus  we  conclude  that  WorldApp  Key  Survey 
does  not  meet  ARFs  technical  requirements.  Of  the  two  products  left  in  consideration  (i.e., 
Confirmit  EFM  Professional  and  Vovici  EFM  Community),  Confirmit  EFM  Professional  has 
more  available  features  that  suit  the  needs  of  ARI  researchers,  but  Vovici  EFM  Community  has  a 
Certificate  of  Networthiness  for  an  earlier  version  of  the  current  product.  Both  products’  IT 
features  should  be  easily  supported  by  ARI,  and  the  company  and  support  information  suggest 
that  both  companies  have  the  size  and  health  suitable  to  a  long-standing  company.  Based  on  the 
overall  market  analysis,  we  conclude  that  both  Confirmit  EFM  Professional  and  Vovici  EFM 
Community  are  the  products  which  best  meet  ARFs  needs  and  technical  requirements. 
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COST  ANALYSIS 


ICF  conducted  a  cost  analysis  in  which  the  detailed  costs  estimates  were  gathered  and 
projected  for  the  four  finalist  COTS  Web-based  survey  software  products.  The  methodology  and 
results  of  the  cost  analysis  are  summarized  below. 

Methodology 

The  methodology  employed  for  the  cost  analysis  included  Internet  searches  and 
telephone  meetings  with  software  vendors.  This  cost  analysis  included  the  following  three 
sub  tasks: 

1.  Develop  software  configuration  specifications  for  ARFs  needs,  including  optional 

packages, 

2.  Request  estimates  from  vendors,  and 

3.  Review  estimates  and  revise  as  necessary. 

The  methodology  of  each  cost  analysis  subtask  is  detailed  below. 

Develop  Software  Configuration  Specifications  for  ARI's  Needs,  Including  Optional  Packages 

ICF  developed  specifications,  based  on  focus  groups  and  individual  interviews  with  ARI 
staff  and  contractors,  which  describe  ARFs  online  survey  software  needs  to  provide  to  COTS 
survey  software  product  vendors.  Such  specifications  included  the  costs  of  initial  licensing, 
installation  assistance,  yearly  support  and  maintenance  fees,  advanced  add-on  features,  custom 
services,  and  training.  Estimated  costs  were  based  on  the  following  specifications: 

■  At  least  5  developer  licenses, 

■  At  least  5  reporting  licenses, 

■  Unlimited  number  of  surveys, 

■  At  least  60,000  completed  survey  responses  per  year, 

■  A  vendor  representative's  assistance  with  installation  on  ARFs  servers, 

■  1  paper  training  guide,  and 

■  1  day  of  private  training  at  ARI  headquarters  in  Arlington,  VA. 

Software  configuration  specifications  and  the  associated  costs  for  each  product  are  found  in 
Appendix  K. 

Request  Estimates  from  Vendors 

ICE  evaluation  team  members  provided  the  survey  software  configuration  specifications 
to  each  of  the  software  vendors  whose  products  were  tested  and  evaluated.  ICE  requested  a  cost 
estimate  from  each  vendor  for  the  costs  for  initial  licensing,  license  renewals,  advanced  features, 
software  modifications,  and  training. 
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Review  Estimates  and  Revise  as  Necessary 


The  evaluation  team  members  compiled  and  documented  each  cost  estimate  received  (see 
Appendix  K).  Costs  for  additional  items  listed  on  estimates  (e.g.,  private  labeling)  were 
requested  from  the  other  vendors.  The  cost  estimates  for  each  vendor  were  revised  as 
appropriate. 


Results 

The  cost  analysis  revealed  that  products  range  widely  from  an  average  of  $15,000  per 
year  to  an  average  of  $54,000  per  year  (over  a  5-year  period).  Costs  were  estimated  based  on 
expected  needs  for  the  C'  year  after  purchase  and  total  cost  for  5  years  after  purchase  (results 
displayed  in  Table  7).  Over  a  5-year  period,  Vovici  EFM  Community  is  estimated  to  cost  the 
least  (i.e.,  $75,975),  while  Confirmit  EFM  Professional  is  estimated  to  cost  the  most  (i.e., 
$264,075). 

Detailed  cost  breakdowns  for  each  finalist  product  appear  in  Appendix  M.  The  estimates 
provided  are  for  informational  purposes  only.  It  is  worth  noting  that  software  vendors  may 
discount  product  prices  when  multiple  years’  worth  of  software  is  ordered. 

Table  7 


Web-Based  Survey  Software  Product  Costs 


Survey  Vendor 

Confirmit  EFM 
Professional 

Vovici  EFM 
Community 

World  App 
Key  Survey 

Zarca 

Interactive 

Estimated  1st- 
year  cost*: 
Estimated 

$58,075 

$18,995 

$49,029 

$102,000 

5-year  total 
cost*: 

$274,075 

$75,975 

$78,945 

$162,000 

*  Estimated  costs  include  at  least  5  developer  licenses,  at  least  5  reporting  licenses,  unlimited  number  of  surveys,  at 
least  60,000  completed  survey  responses  per  year,  a  vendor  representative's  assistance  with  installation  on  ARI's 
servers,  1  paper  training  guide,  and  1  day  of  private  training  at  ARI. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


ICE  analyzed  ARI’s  survey  software  needs  and  reviewed  the  extent  to  which  74  COTS 
Web-based  survey  software  products  would  meet  those  requirements.  Based  on  our  technical 
review  and  analyses,  we  conclude  that  Confirmit  EEM  Professional  and  Vovici  EEM 
Community  best  meet  ARI’s  research  needs  and  technical  requirements. 

The  market  and  cost  analyses  revealed  that  Confirmit  EEM  Professional  offers  a  higher 
percentage  of  desired  survey  features  (81.58%)  than  Vovici  EEM  Community  and  is  compatible 
with  the  information  technology  (IT)  and  security  requirements  of  ARI.  Confirmit  EEM 
Professional,  however,  is  the  most  expensive  product  reviewed,  with  an  estimated  cost  of 
$274,075  over  five  years.  Vovici  EEM  Community  has  a  Certificate  of  Networthiness  for  an 
earlier  version  of  the  current  product  and  is  a  more  cost-effective  solution,  with  an  estimated  cost 
of  $75,975  over  five  years.  Vovici  EEM  Community,  however,  has  a  slightly  lower  percentage  of 
survey  features  available  (72.37%)  than  Confirmit  EEM  Professional.  It  is  important  to  note  that 
these  recommendations  are  based  on  the  current  requirements  of  ARI  and  the  current  COTS 
Web-based  survey  software  features  and  infrastructures.  Because  the  Web-based  survey  software 
industry  is  dynamic  and  constantly  evolving,  ARI  should  reevaluate  their  Web-based  survey 
software  product  on  an  as-needed  basis. 
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Appendix  A 
Needs  Analysis 

Desired  Functions  and  Features  List  Categories 
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Survey  Software  Functions  and  Features 


USAGE  LIMITS: 

o  Maximum  number  of  active  surveys  in  the  field  at  any  one  time 
o  Maximum  respondents  per  survey 
o  Maximum  survey  size  (#  of  questions) 


DEVELOPER  EEATURES: 

o  Number  of  simultaneous  developers  in  program  at  one  time 
o  User  roles  (e.g.,  survey  developer,  report  viewer,  etc.) 
o  Support  needed  (e.g.,  phone,  email,  EAQ,  online  manual) 


SYSTEM  STRUCTURE: 

o  Real-time  (while  survey  is  in  the  field)  access  to  data  through  interface 
o  Documentation/Training 

o  Integration  options  (e.g.,  integrate  with  personnel  data) 


SURVEY  QUESTIONNAIRE: 

o  Eile  export  format  (e.g..  Word) 

o  Eile  import  capability  (e.g.,  import  a  Word  document  to  get  question  text  in) 
o  Non-standard  Item  types 
o  Complex  Routing 
o  Data  piping 

o  Persistence  (e.g.,  cookie,  login) 
o  Authentication  (e.g.,  username/password) 


MAIL  CAMPAIGN: 

o  Customizable  "Prom"  address 
o  Customizable  "Reply  To"  address 

o  Type(s)  of  link  to  survey  in  email  (e.g.,  URL,  link  with  URL  hidden,  embedded 
passwords  in  URL) 

o  HTML  format  for  email  (as  opposed  to  only  text  format) 
o  Type  of  participant  tracking  (e.g.,  started,  completed,  abandoned) 
o  Number  of  reminders 


REPORTING: 

o  Online  graphs 
o  Online  tables 
o  Report  export  formats 
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SCORING: 

o  Scale  scores 
o  Complex  scoring 


DATA: 

o  Data  export  formats 
o  Data  export  filtering 


OTHER 
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Appendix  B 

Needs  Analysis 
Focus  Group  Protocol 
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Focus  Group  Protocol  for  Needs  Analysis 

I.  Introduction 

Thank  you  for  joining  us  today.  My  name  is _ and  this  is _ .  We  work  for 

IGF  International,  which  is  the  research  firm  hired  to  evaluate  and  recommend  a  survey  software 
platform  that  best  serves  ARFs  needs. 

As  part  of  this  focus  group,  we  would  like  to  talk  with  you  about: 

•  The  pros  and  cons  of  Raosoft,  your  current  survey  software  AND 

•  Your  needs  with  regard  to  an  online  survey  software  platform. 

Although  we  will  be  taking  detailed  notes  on  the  information  collected  in  this  focus  group,  all 
data  will  remain  anonymous.  That  is,  we  will  not  link  your  name  or  demographic  information 
with  any  comments  that  you  make.  The  results  of  the  focus  groups  will  be  combined  with  the 
wish  lists  you  provided  to  identify  criteria  against  which  we  will  evaluate  many  Web-based 
survey  software  platforms. 

There  are  several  ground  rules  for  this  session  to  help _ taking  notes: 

•  Speak  clearly  and  one  at  a  time 

•  There  are  no  right  or  wrong  answers 

•  We  want  to  hear  the  good  and  the  bad 

•  We  respect  and  value  differences  of  opinion 

•  Please  avoid  sidebar  conversations 

Are  there  any  questions  before  we  begin? 

II.  Focus  Group  Questions 

Criticality/Clarification  of  Wish  List  Features  [take  at  most  40  minutes  discussing  this 
section] 

1.  In  question  development,  how  much  control  is  necessary  in  designing  questions  (e.g., 
matrix  questions)? 

Do  you  need  to  get  into  the  HTML  code  to  augment  the  standard  offerings  of  the 
COTS  software  (e.g.,  add  java  script  to  do  complex  scoring  that  will  determine  skip 
patterns)? 

2.  How  important  is  it  to  be  able  to  create  online  help  features,  such  as  an  electronic  help 
document  for  participants  to  reference  or  pop-up  boxes  for  certain  questions  that  display 
definitions? 

3.  It  was  mentioned  in  the  wish  lists  that  you  would  like  the  ability  to  create  parts  of  survey 
in  different  files  and  combine  later  into  one  survey.  Would  this  take  the  form  of  a  survey 
library  that  would  include  partial  surveys  that  can  be  combined  into  one? 
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If  the  application  has  a  Word  document  import  function,  would  it  be  sufficient  to 
combine  Word  documents  and  upload  the  combined  survey  (i.e.,  create  the  survey 
outside  of  the  application  and  then  import  it). 

4.  Where  in  the  survey  is  it  necessary  to  be  able  to  add  graphics  (e.g.,  header,  question  text, 
response  option)? 

5.  We  are  trying  to  assess  the  level  of  conditional  logic  and  branching  you  need  for  surveys. 
Please  describe  an  example  of  the  most  complicated  branching  pattern  you  may  need. 

6.  In  terms  of  exporting  the  survey  to  Word  or  PDF,  how  close  do  you  want  the  export  to  be 
to  a  final  paper  version? 

Would  you  like  to  revise  the  survey  once  it  is  exported? 

How  important  is  it  is  to  export  survey  formatting  (e.g.,  template)  in  addition  to 
exporting  question  and  response  option  text? 

7.  What  functionality  would  you  need  in  the  testing  phase  (i.e.,  after  survey  development 
before  survey  launch)? 

Is  it  adequate  to  be  able  to  publish  the  survey  (i.e.,  place  it  on  the  Web),  test  it, 
and  clear  all  test  data  prior  to  giving  out  survey  URL? 

If  not,  please  describe  this  function  in  its  ideal  form. 

8.  How  often  do  you  need  to  change  survey  features  after  the  survey  is  in  the  field? 

Do  you  need  the  ability  to  change  survey  question  text  while  the  survey  is  in  the 
field? 

Do  you  need  the  ability  to  add  response  options  while  the  survey  is  in  the  field? 

9.  What  type  of  participant  tracking  is  required? 

Do  you  need  an  application  that  records  a  unique  ID  for  each  case  (e.g.,  IP  address  or 
some  other  way  to  link  back  to  a  PC)? 

Do  you  need  the  ability  to  tell  exactly  which  survey  question  was  the  last  question  the 
respondent  answered  (track  where  survey  was  abandoned)? 

10.  How  important  is  it  for  a  survey  software  platform  to  do  data  cleaning  and  analyses 
within  the  application  (as  opposed  to  being  exported  and  all  analyses  and  report 
development  done  in  an  outside  program)? 

Do  you  need  to  recode  responses  within  the  application? 
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Do  you  need  to  conduct  statistics  (e.g.,  mean  median,  chi  square,  standard 
deviation)  within  the  application? 

Would  you  use  the  results  of  these  calculations  within  the  survey  (e.g.,  to 
determine  skip  patterns  or  response  options  shown  in  a  question)? 

1 1 .  How  important  is  it  for  a  survey  software  platform  to  do  advanced  reporting  within  the 
application  (as  opposed  to  being  exported  and  reporting  being  done  in  an  outside 
program)? 

Do  you  need  to  customize  the  reporting  within  the  application  to  display 
frequency  tables,  cross-tab  tables,  bar  charts,  pie  charts? 

At  what  level  do  you  need  to  customize  reports  within  the  application? 
o  Only  use  templates  provided  by  the  application 
o  Make  minor  color  or  font  changes 
o  Edit  all  aspects  of  an  exhibit 

How  important  is  it  for  you  to  export  the  report  into  Microsoft  PowerPoint  or 
Excel? 

12.  Are  there  additional  absolutely  critical  functions  that  are  not  standard  on  survey 
software  programs  and  were  not  discussed  or  were  not  on  the  questionnaire? 

13.  How  important  are  the  following  features  over  some  of  the  previous  survey  development 
features  we  have  discussed?  [list  features  we  see  as  potentially  limiting/problematic] 

o  _ 

o  _ 

o  _ 

o  _ 


Pros/Cons  of  Raosoft/Interform  [leave  at  least  30  minutes  for  this  section] 

We  are  interested  to  hear  about  the  best  and  worst  features  of  your  current  software 
platforms,  Raosoft  or  Interform.  I  will  be  asking  you  a  series  of  questions  about  the  cons  and 
then  the  pros  of  your  current  software.  When  answering  each  question,  please  state  which 
software  system  you  use. 

Cons 

14.  Have  you  had  any  specific  trouble  with  Raosoft  or  Interform  with  regard  to  usage  limits 
(e.g.,  maximum  number  of  active  surveys,  questions,  or  respondents)? 

15.  Are  there  any  notable  features  missing  with  Raosoft’ s  or  Interform’s  developer  features 
or  system  structure  (e.g.,  number  of  developers  at  a  time,  access  to  data,  documentation 
and  support)? 
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16.  Are  there  any  notable  features  missing  with  Raosoft’s  or  Interform’s  survey  questionnaire 
(e.g.,  file  export,  item  types,  branching,  piping,  persistence)? 

17.  Are  there  any  notable  features  missing  with  Raosoft’s  or  Interform’s  email  distribution 
and  tracking? 

18.  Are  there  any  notable  features  missing  with  Raosoft’s  or  Interform’s  data  analysis, 
scoring,  or  reporting? 

19.  Are  there  any  notable  features  missing  with  Raosoft’s  or  Interform’s  data  exporting? 

20.  Are  there  any  other  notable  features  in  which  you  struggled  with  Raosoft  or  Interform, 
significantly  impacting  your  ability  to  do  quality  surveys? 

Pros 

21.  Are  there  any  notable  features  with  regard  to  usage  limits  (e.g.,  maximum  number  of 
active  surveys,  questions,  or  respondents)  that  Raosoft  or  Interform  does  particularly  well 
and  you  would  like  to  see  in  a  different  survey  software  platform? 

22.  Are  there  any  notable  features  that  Raosoft  or  Interform  does  well  with  regard  to 
developer  features  or  system  structure  (e.g.,  number  of  developers  at  a  time,  access  to 
data,  documentation  and  support)? 

23.  Are  there  any  notable  features  that  Raosoft  or  Interform  does  well  with  regard  to  the 
survey  questionnaire  (e.g.,  file  export,  item  types,  branching,  piping,  persistence)? 

24.  Are  there  any  notable  features  that  Raosoft  or  Interform  does  well  with  regard  to  email 
distribution  and  tracking? 

25.  Are  there  any  notable  features  that  Raosoft  or  Interform  does  well  with  regard  to  data 
analysis,  scoring,  or  reporting? 

26.  Are  there  any  notable  features  that  Raosoft  or  Interform  does  well  with  regard  to  data 
exporting? 

27.  Are  there  any  notable  features  that  Raosoft  or  Interform  does  well  that  are  critical  to  your 
ability  to  do  quality  surveys? 

II.  Conclusions 

Thank  you  for  taking  the  time  to  provide  input  to  this  important  project.  We  will  do  our  best  to 
incorporate  your  comments  from  today’s  meeting  and  the  survey  results  to  choose  a  software 
provider  with  as  much  advanced  functionality  to  meet  your  needs.  If  you  think  of  any  other 
comments  or  suggestions  you  would  like  to  provide,  you  can  call  or  email  me  to  provide  more 
input.  [Pass  out  Beth ’s  business  card]. 
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Appendix  C 
Needs  Analysis 

Focus  Group  Questionnaire  on  Feature  Criticality 
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Name 


~7i 

/jr 

7f 

/o 

2 

Group 

o 

o 

'o| 

/ /t/i/i 

m/M/ 

W/IL 


The  ability  to  add  to  and  select  from  a  question  library  of  full 
questions  (question  text  and  response  options) 

■c 

■c 

■c 

■O' 

TT 

The  ability  to  add  to  and  select  from  a  response  opbon  library  of 
commonly-used  response  option  sets 

IT 

"O' 

"C 

"C 

System  Structure 

PuSlish  a  survey  to  a  transportable  file  that  can  be  run  on  local 
machines  (e  g  .  non-Internet) 

Ability  to  withdraw  a  survey  (either  one  that  has  been  electronically 
submitted  or  one  that  is  partially  complete) 

Mult-language  capability _ 

System  automatically  creates  an  archive''bacl<-up  of _ 

System  allows  for  field.’telephone  interviewing 

System  keeps  historical  log  files  (i  e  .  that  record  upload,  deletion. 

and  emails  to  survey  respondents)  that  cannot  be  deleted  by  ARI 


/ /i/f/// 

/ 

MW// 


Ability  to  automatically  create  and  assign  IDs/passwords 

O 

o 

O 

o 

o 

The  ability  to  filter  reminder  email  recipients  (e.g  .  send  to  specific 
sub-samples  based  on  rank) 

IT 

75" 

TT 

75" 

The  ability  to  remove  a  respondent  from  the  survey  distribution  list 
so  they  do  not  receive  future  correspondence  (e.g..  reminder  email) 

■c 

TT 

TT 

75" 

TT 
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Itoadi  «m^Mi 


[Data  ExpoftinoTT racking 


Ability  to  export  data  directly  to  SPSS 

0 

O 

0 

o 

o 

Abilitv  to  export  data  directly  to  SAS 

IT 

0 

0 

0 

0 

Ability  to  export  data  directly  to  Microsoft  Access 

0 

o 

0 

o 

o 

Ability  to  store  the  data  in  an  encrypted  format 

0 

o 

o 

o 

o 

Ability  to  assipn  export  variable  names 

IT 

0 

0 

0 

0 

Ability  to  assign  export  response  option  values 

0 

o 

0 

0 

o 

Ability  to  track  the  exact  question  the  participant  answered  prior  to 

O 

o 

o 

o 

o 

abandoning  the  survey 

Ab  lity  to  print  out  the  question  properties  for  the  survey  (e.g..  page, 
question  variable  name,  question  text,  and  question  scale) 

IT 

Ab  lity  to  have  different  types  of  response  options  in  the  same  matrix 
question 

0 

O 

0 

O 

0 

Automatic  numbering  with  the  opton  of  displaying  numbers  or  not 

75" 

0 

0 

0 

0 

Aovaneed  Formatting  (e.g..  Irie  spacing,  line  indent,  column  labels, 
li  ne  breaks,  ■•esponse  column  widths: 

0 

o 

0 

o 

o 

Advanced  Page  Layout  (e.g  .  resize  objects,  layer  objects,  snap 
objects  to  a  gnd.  and  lock  a  page) 

0 

o 

0 

o 

o 

Customizable  templates  (e  g.,  theme  or  cascading  style  sheets) 

17 

0 

0 

0 

0 

Ab  lity  to  add  HTML  code  to  question  text  and'or  question 

0 

0 

0 

o 

o 

Ab  litv  to  name  (label  i  pages 

17 

0 

0 

0 

0 

Ab  litv  to  name  (label  )  questions 

0 

o 

0 

o 

0 

Ab'lity  to  edit  navigaton  button's  label 

0 

o 

0 

o 

o 

Ab  lity  to  show  previous  answers  in  the  text  of  current  questions  (i.e. 
piping  into  queston  text) 

17 

0 

0 

0 

0 

Ab'lity  to  show  only  the  response  options  that  were  selected  (or  not) 
in  a  previous  question  (i.e.  piping  into  response  options) 

0 

o 

0 

o 

o 

Ab  lity  to  skipi'ask  particular  questions  based  upon  record 
data'demographc  data  (i  e.  piping  data  from  p'eioaded 

'XT 

0 

0 

0 

0 

Ab  lity  to  randomize  the  order  of  response  options 

0 

o 

0 

o 

0 

Ab  litv  to  randomize  the  order  of  Questions 

17 

0 

0 

0 

0 

Ab  lity  to  randomize  the  order  of  pages 

0 

o 

0 

o 

0 

Automatically  add  response  values  as  each  response  is  added  (e.g.. 
summing  percents) 

0 

0 

0 

o 

o 

Ab  lity  to  writeicode  survey  branching  logic  (rather  than  only  having 
the  option  to  use  a  drop  down  I'St) 

0 

o 

0 

o 

0 

Offer  respondents  an  easy  way  to  review  their  answers  to  previous 
questions  (i.e.  without  having  to  backtrack  through  every  question) 

17 

0 

0 

0 

0 

Qualitative  data  analysis  functionality  (e.g  .  keying  in  on  words  or 
phases) 

17 

0 

0 

0 

0 

CF  titamMDMl 
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Appendix  D 


Information  Assurance  Review 
Survey  Software  IT  and  Security  Requirements 
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Table  D-1 

Information  Assurance  Review:  Survey  Software  Requirements 


Reg  Section  Title 


Text 


AR 

25-1 


5-3 


Information 

system 

certification/ 

accreditation 


c.  NETCOM  has  overall  responsibility  for  ensuring  that  all 
information  systems  are  properly  certified  and  accredited  in 
accordance  with  the  DITSCAP.  MACOMs,  PEOs,  and  direct 
reporting  PMs  will  be  responsible  for  certification  and 
accreditation  of  MACOM,  PEO,  and  direct  reporting  PM 
unique  systems  that  they  own  and  operate.  Tenant  lAMs  are 
responsible  for  ensuring  that  tenant  information  systems  are 
certified  and  accredited  for  that  tenant  organization.  The 
DITSCAP^  will  be  applied  to  all  systems  requiring 
certification  and  accreditation  throughout  their  life  cycle.  (See 
also  AR  25-2  and  DoDI  5200.40.)  Where  applicable,  all  lA- 
related  Government-off-the-shelf  (GOTS)  and  COTS 
hardware,  firmware,  and  software  components  and  IT 
products  used  in  the  Army  Information  Infrastructure  must  be 
evaluated  and  acquired  in  accordance  with  the  National 
Security  Telecommunications  Information  Systems  Security 
Policy  (NSTISSP)  No.  11,  and  other  applicable  national  and 
DoD  policy  and  guidance  identified  in  this  chapter  or  in  AR 
25-2. 


a.  Controls  will  be  implemented  to  protect  system  software 
from  compromise,  subversion,  or  tampering.  The  installation 
lAM,  Configuration  Management  Board,  Configuration 
AR  ^  ^  Software  Control  Board,  and  designated  approval  authority  (DAA) 

25-1  security  must  approve  all  software  used  on  Army  networks  prior  to 

installation  and  operation. 


AR  ,  ^  Software 
25-1  security 


b.  When  database  management  systems  (DBMS)  containing 
classified  defense  information  are  used,  the  classified 
identifiable  element  (for  example,  word,  field,  or  record) 
within  the  database  must  be  protected  according  to  the  highest 
security  classification  of  any  database  element.  If  the  database 
cannot  provide  field  protection,  then  it  should  provide  record 
protection  to  the  highest  security  classification  level  of  the 
fields  within  the  record.  Database  systems  that  do  not  provide 
protection  at  the  record  or  field  level  will  be  restricted  to 
operation  in  the  dedicated  or  system  high  security  mode.  In  all 
cases,  the  DBMS  must  meet  the  minimum  trust  requirements. 
(Eor  more  information,  refer  to  AR  25-2.) 


IT  support 
AR  ,  ^  .  principles: 
25-1  Product 

ownership. 


The  products  of  Army-related  work  are  the  property  of  the 
U.S.  Government,  regardless  of  the  ownership  of  the 
automation  hardware  or  software. 


^  DITSCAP:  Defense  Information  Technology  Security  Certification  and  Accreditation  Process 
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Summary  of 
Relevant 
Material 


The  highest 
security  level  of 
ARI  survey  data 
is  confidential  and 
so  is  not  subject  to 
regulation  beyond 
AR  25-1,  AR  25- 
2,  and  DoDI 
5200.40. 


IMCEN  must 
approve  all 
software  installed 
on  its  servers. 


The  highest 
security  level  of 
ARI  survey  data 
is  confidential, 
and  so  the  DBMS 
must  be  protected 
according  to 
IMCEN  standards 
for  this  type  of 
data. 


All  products  of 
the  software  (i.e. 
surveys)  are  the 
property  of  the 
U.S.  Government. 


License 

agreement  cannot 
include  any 
ownership  of  the 
products  by  the 
software  vendor. 


6-2  g. 


6-2  g. 


IT  support 
principles: 
Installation- 
level  technical 
support  and 
service. 


Computing 

services: 

Standard 

software 

applications. 


Computing 

services: 

Standard 

software 

applications. 


(2)  Since  DOIMs  cannot  provide  equal  technical  support  (for 
example,  troubleshooting  and  training  for  all  COTS  hardware 
and  software  products),  lists  of  supported  products  may  be 
promulgated  that  restrict  the  scope  of  support  to  the  listed 
products.  In  establishing  such  lists  and  levels  of  support, 
installations  will  not  restrict  the  use  of  the  common 
infrastructure  of  any  JTA-A-compliant  information  system. 
The  lists  will  not  be  used  as  the  justification  for 
eliminating  competition  in  contracting.  Supported 
organizations  and  IT  fielding  organizations  that  rely  on 
common  network  capabilities  may  deviate  from  supported 
product  lists  on  an  exception  basis  only. 


(5)  COTS  products  or  existing  GOTS  software  applications 
will  be  preferred  to  funding  new  application  development. 

The  suitability  of  COTS  or  GOTS  applications  for  satisfying 
operational  requirements  will  be  evaluated  prior  to  initiating  a 
development  effort.  Evaluation  should  include  not  only 
identification  of  COTS  or  GOTS  products  that  can  satisfy 
DoD,  Army,  or  system-specific  requirements,  but  also  an 
assessment  of  the  likelihood  that  the  product  or  subsequent 
versions  of  the  product  will  be  available  and  supported 
throughout  the  life  cycle  of  the  system. 


(6)  Software  applications  will  be  reviewed  at  system 
milestone  reviews.  The  review  will  be  based  on  a  business 
case  that  considers  information  exchange  requirements  and 
cost  effectiveness  as  viewed  from  an  Army-wide,  not 
individual  system,  perspective.  At  a  minimum,  software 
applications  will  be  designed  to — 

(a)  Permit  users  to  access  shared  data  in  a  consistent 
standards -based  approach,  independent  of  specific  vendors’ 
IT. 

(h)  Be  independent  of  vendor-specific  data  management  and 
access  schemes. 

(c)  Provide  users  with  transparent  access  to  nonlocal  data. 

(d)  Permit  use  of  data  and  information  as  Army-wide  assets. 


The  software 
chosen  does  not 
need  to  be 
supported  by 
IMCEN  (in  terms 
of  training  and 
trouble  shooting) . 


Preference  is  to  be 
given  to  software 
that  will  be 
supported  and 
continue  to  be 
upgraded  (i.e. 
preference  should 
be  given  to  a 
vendor  that  has  a 
strong  likelihood 
of  existing 
throughout  the 
lifecycle  of  ARI’s 
survey  software 
need.) 


Data  should  be 
stored  in  a  format 
independent  of 
software  vendor. 


AR  6-2  g.  Computing 


(7)  Use  standard  data  formats  as  approved  for  use  by  the  DoD  Data  should  be 
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Section  Title 
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Summary  of 
Relevant 
Material 

25-1 

services: 

Standard 

software 

applications. 

Net-Centric  Data  Management  Program  described  in  chapter 

4  of  this  regulation. 

accessible  in  a 
standard  format 
(i.e.  one  approved 
by  the  IMCEN). 

(a)  The  Defense  Supplement  to  the  Federal  Acquisition 
Regulation  (DFARS)  subpart  208.74,  requires  DoD 
components  to  purchase  from  the  DoD  inventory  before 
Computing  buying  the  product  from  another  source.  When  an  activity 

services:  Ojfice  requires  a  COTS  product,  the  supporting  DOIM  will  Preference  should 

AR  g  2  g  automation.  determine  if  it  is  available  under  the  DoD  Enterprise  Software  be  given  to  COTS 

25-1  ^  Enterprise  Initiative  (ESI).  Enterprise  software  agreements  (ESA)  already  in  DoD 

software  negotiated  with  specific  software  publishers  or  their  agents  inventory. 

licenses.  provide  the  best  available  prices,  terms  and  conditions.  The 

DoD  ESA  is  the  DoD  implementation  of  the  Eederal-wide 
SmartBUY  program. 


Computing 
services:  Office 

AR  ,  -  automation. 

25-1  (  )  Enterprise 

software 
licenses. 


(c)  The  DOIMs  will  coordinate  their  acquisition  plans  with 
the  ASCPO^  concerning  specific  products  prior  to  entering 
into  an  agreement  with  any  COTS  vendors.  If  the  existing 
ESA  does  not  contain  the  desired  terms  and 
conditions  or  prices,  the  DOIM  must  notify  the  ESA  manager 
so  that  the  manager  may  improve  the  existing  ESA  prior  to 
the  DOIM’s  executing  any  other  agreement.  The  ASCPO  is 
responsible  for  authorizing  new  ESA  agreements  and  for 
granting  waivers  for  Army  activities  to  acquire  an  ESI- 
managed  COTS  product  from  any  other  source.  The  ASCPO 
Web  site  is  http://pmscp.monmouth.army.mil.  See  also  the 
DoD  ESI  homepage,  which  lists  all  ESI-managed  software: 
http://www.don-imit.navy.mil/esi/. 


Coordinate 


purchase  with 
ASCPO. 


AR 

25-1 


Computing 
services:  Life- 
6-2  m.  cycle 

depreciation. 


In  planning  life-cycle  requirements  and  calculating  economic 
benefits  of  automation  IT,  3  years  from  the  initial  date  of 
installation  will  be  used  as  the  metric  for  obsolescence  of 
common-use  IT.  Serviceability,  maintainability,  and  utility 

will  also  be  used  as  factors  to  consider  in  specific  life -cycle  Preference  should 
replacement  decisions.  This  metric  may  vary  according  to  be  given  to  COTS 

mission  requirements.  System  planning  should  include  products  that 

provisions  for  product  upgrades  during  the  projected  life  span  include  upgrades 
to  cover  potential  obsolescence,  lack  of  vendor  support,  for  at  least  3  years 

support  of  information  assurance  and  requirements,  and  after  purchase, 

incorporation  of  alternative  products  or  technologies  when 
such  changes  are  justifiable  and  cost  effective. 


AR 

25-1 


6^  n. 


Telecommunica  (2)  AKO  (www.us.army.mil)  is  the  enterprise  portal  for  Army  Authentication 
tions  systems  unclassified  intranets  and  the  NIPRNET.  AKO  is  the  single  must  be  achieved 

and  services:  Army  portal  for  authenticating  users  to  gain  access  to  Army  through  AKO 


*  Army  Small  Computer  Program  Office 
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Reg  Section 


Title 


Text 


Summary  of 
Relevant 
Material 

Internet  (World  systems  and  Web  servers.  Existing  Army  portals  or  Web  LDAP  or  a  waiver 

Wide  Web  servers  with  authentication  services  will  migrate  must  be  granted. 

(WWW)),  authentication  support  to  AKO  unless  waived  by  CIO/G-6. 

intranets,  and  The  AKO-S  is  the  enterprise  portal  for  classified  intranets 

extranets.  and  the  SIPRNET.  The  use  of  AKO  and  AKO-S  enables 

optimal  sharing  of  Army  information  and  knowledge 
resources  across  the  entire  Army  enterprise.  Army  activities 
will  maximize  their  use  of  AKO  resources,  features,  and  tools 
in  order  to  reduce  the  need  for  installation  and  MACOM 
investment  in  the  same  types  of  IT  resources. 

(a)  Army  Web-enabled  business  applications  are  required  to 
be  linked  to  the  AKO  portal.  Initial  minimum  standard  to  link 
applications  to  AKO  is  a  URL  link  on  the  Army  portal.  The 
objective  standard  to  link  applications  to  AKO  is  to  use  the 
AKO  directory  services  for  authentication  as  well  as  a  URL 
link  on  the  Army  portal. 

(b)  Proponents  are  required  to  establish  the  appropriate 
mechanisms  to  protect  sensitive  information  from  being 
accessed  by  unauthorized  individuals.  AKO  is  responsible  for 
generating  user  IDs  and  accounts,  performing  authentication 
via  secure  Lightweight  Directory  Access  Protocol  (LDAP) 
directory  services,  publishing  updates  to  the  technical 
mechanism  used  for  directory  services,  and  incorporating 
appropriate  security  measures.  All  applications.  Web  sites. 


(8)  Organizations  requiring  private  Web  sites  (for  example, 
intranets,  extranets)  must  register  them  with  the 
NETCOM/9th  ASC  Theater  Network  Operations  and  Security 
Center  (TNOSC)  and  assure  that  the  secure  sockets  layer 
(SSL)  is  enabled  and  that  PKI  encryption  certificates  are 
loaded.  Use  of  Internet  protocol  restriction  by  itself  is 
insufficient;  such  sites  will  be  considered  publicly  accessible 
rather  than  private.  PKI  Web  server  certificates  may  be 
Telecommunica  obtained  from  the  NETCOM/9th  ASC  TNOSC.  SSL  must  be 

tions  systems  (a)  All  Web  applications  will  use  AKO  LDAP  to  authenticate  enabled  using 

and  services:  clients,  unless  waived  by  NETCOM/9th  ASC.  DoD  PKI 

AR  6^  n  Internet  (World  (h)  All  unclassified,  private  Army  Web  servers  will  be  certificates  for 

25-1  '  Wide  Web  enabled  to  use  DoD  PKI  certificates  for  server  authentication  server 

(WWW)),  and  client/server  authentication.  The  following  type  of  Web  authentication  and 

intranets,  and  server  is  exempt  from  this  mandate:  any  unclassified  Army  client/server 

extranets.  Web  server  providing  nonsensitive,  publicly  releasable  authentication, 

information  resources  categorized  as  a  private  Web  server 
only  because  it  limits  access  to  a  particular  audience  only  for 
the  purpose  of  preserving  copyright  protection  of  the 
contained  information  sources,  facilitating  its  own 
development,  or  restricting  access  to  link(s)  to  limited  access 
site(s)  (and  not  the  information  resources). 


D-5 


Reg  Section 


Title 


Text 


Summary  of 
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AR 

25-1 

6^  n. 

Telecommunica 
tions  systems 
and  services: 
Internet  (World 
Wide  Web 
(WWW)), 
intranets,  and 
extranets. 

(9)  To  ensure  ease  of  access,  public  Web  sites  that  collect 
sensitive  but  unclassified  information  from  the  general  public 
as  part  of  their  assigned  mission  are  authorized  to  use 
approved  commercially  available  certificates  to  provide  SSL 
services.  Select  from  the  trusted  and  validated  products  lists 
on  DISA’s  Web  site  (http://iase.disa.mil/common/index.html). 

The  minimum 
security  provided 
for  sensitive  but 
unclassified 
information  is 

SSL. 

AR 

25-2 

4-5  c. 

Minimum 

information 

assurance 
requirements: 
Access  Control 

(4)  Verify  systems  are  configured  to  automatically  generate 
an  auditable  record  or  log  entry  for  each  access  granted  or 
attempted. 

Audit  log  must  be 
available  for  the 
development  of 
the  survey  and  for 
the  fielding  of  the 
survey. 

AR 

25-2 

4-5  c. 

Minimum 

information 

assurance 
requirements: 
Access  Control 

(5)  Validate  that  systems  identify  users  through  the  user’s  use 
of  unique  user  identifications  (USERIDs). 

Authentication  of 
developers  must 
include  a  unique 
identifier. 

AR 

25-2 

4-5  c. 

Minimum 

information 

assurance 
requirements: 
Access  Control 

(6)  Validate  that  systems  authenticate  users  through  the  use  of 
the  CAC  as  a  two-factor  authentication  mechanism.  The  CAC 
has  certificates  on  the  integrated  circuit  chip  (ICC),  and  will 
be  used  as  the  primary  user  identifier  and  access  authenticator 
to  systems. 

Authentication  of 
developers  should 
at  a  minimum  be  a 
two-factor 
mechanism. 

AR 

25-2 

4-5  c. 

Minimum 

information 

assurance 
requirements: 
Access  Control 

(7)  Validate  system  configurations  to  authenticate  user  access 
to  all  systems  with  a  minimum  of  a  USERID  and  an 
authenticator  when  the  systems  are  incapable  of  CAC 
enablement  until  these  are  replaced.  An  authenticator  may  be 
something  the  user  knows  (password),  something  the  user 
possesses  (token),  or  a  physical  characteristic  (biometric).  The 
most  common  authenticator  is  a  password. 

Authentication  of 
developers  should 
at  a  minimum  be  a 
two-factor 
mechanism. 

AR 

25-2 

4-5  c. 

Minimum 

information 

assurance 
requirements: 
Access  Control 

(9)  Validate  that  system  configurations  prohibit  anonymous 
accesses  or  accounts  (for  example,  Studentl,  Student2, 

Patronl,  Patron2,  anonymous). 

COTS  should  be 
configurable  to 
not  allow 
anonymous  logins 

AR 

25-2 

4-5  c. 

Minimum 

information 

assurance 
requirements: 
Access  Control 

(12)  Verify  that  system  configurations  generate  audit  logs, 
and  investigate  security  event  violations  when  the  maximum 
number  of  authentication  attempts  is  exceeded,  the  maximum 
number  of  attempts  from  one  IS  is  exceeded,  or  the  maximum 
number  of  failed  attempts  over  a  set  period  is  exceeded. 

COTS  should 
provide  an 
auditable  login 
log  for 
developers. 

AR 

25-2 

4-5  c. 

Minimum 

information 

(13)  Reinstate  accesses  only  after  the  appropriate  lA  (for 
example,  SA/NA)  personnel  have  verified  the  reason  for 

COTS  product 
should  be 
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Title 

Text 

Summary  of 
Relevant 

Material 

assurance  failed  log-on  attempts  and  have  confirmed  the  access-holder’s  configurable  to 

requirements:  identity.  Permit  automatic  account  unlocking,  for  example,  prevent  logins 

Access  Control  after  an  established  time  period  has  elapsed,  as  documented  in  after  rules  set 


the  C&A  package  and  approved  by  the  DAA,  based  on  forth  in  a  pre¬ 
sensitivity  of  the  data  or  access  requirements.  determined 

protocol  have 
been  met. 


AR 

25-2 


4-5  c. 


Minimum 
information 
assurance 
requirements: 
Access  Control 


(14)  If  documented  in  the  C&A  package  and  authorized  by 
the  DAA,  time-based  lockouts  (that  is,  access  is  restricted 
based  on  time  or  access  controls  based  on  IP  address,  terminal 
port,  or  combinations  of  these)  and  barriers  that  require  some 
time  to  elapse  to  enable  bypassing  may  be  used.  In  those 
instances  the  DAA  will  specify,  as  a  compensatory  measure, 
the  following  policies: 

(a)  Implement  mandatory  audit  trails  to  record  all  successful 
and  unsuccessful  log-on  attempts. 

(b)  Within  72  hours  of  any  failed  log-on  and  user  lockout,  lA 
personnel  will  verify  the  reason  for  failure  and  implement 
corrective  actions  or  report  the  attempted  unauthorized  access. 

(c)  The  SA  will  maintain  a  written  record  of  all  reasons  for 
failure  for  1  year. 


COTS  product 
should  be 
configurable  to 
prevent  logins 
after  rules  set 
forth  in  a  pre¬ 
determined 
protocol  have 
been  met. 


AR 

25-2 


4-5  f. 


Minimum 

information 

assurance 

requirements: 

Configuration 

management 

requirements 


AR 

25-2 


4-6 


Controls 


Any  installation 

(4)  Modifying,  installing,  or  downloading  of  any  software  on  of  a  COTS 
any  computer  system  may  affect  system  C&A  and  must  be  product  must  be 

evaluated  and  approved  by  the  lAM  with  the  local  CMB,  coordinated  with 

CCB,  and  DAA.  IMCEN. 


d.  All  COTS  software  used  on  ISs  will  be  fully  licensed 
(under  U.S.  Copyright  Law). 


License 

agreement  must 
be  reviewed  to 
ensure  it  covers 
all  uses  needed  by 
ART 


h.  The  SO  will  place  databases  on  isolated  and  dedicated 
servers  with  restricted  access  controls.  DBAs  will  not  install  The  COTS 
AR  ^  ^  Database  other  vulnerable  servers  or  services  (for  example,  web  product  will  not 

25-2  Management  servers,  ftp  servers)  that  may  compromise  or  permit  use  its  own  server 

unauthorized  access  of  the  database  through  another  critical  application, 
vulnerability  identified  in  the  additional  servers  or  services. 


AR  ^  ^  Database 
25-2  Management 


i.  Databases  should  be  hosted  on  trusted  military  IS  or 
networks.  As  part  of  the  C&A  process,  the  CA  and  DAA  will 
review  and  approve  a  detailed  risk  management  process  as 
documented  in  the  C&A  package  before  operational 
implementation  of  databases  located  in  contractor  owned, 
operated,  or  managed  networks. 


The  database  must 
reside  on  a  trusted 
server. 
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Material 


AR 

25-2 


4-20  e. 

(7) 


Network 
security: 
Protection  of 
internal 
networks 
(Storage  area 
configurations  ) 


(f)  Implement  extranet  connections  through  a  multi-tiered  and 
layered  approach  requiring  separate  and  distinct  servers  across 
the  environment  for  each  tier,  and  minimally  include — 

7.  User  access  tier,  usually  through  a  Web  site  that  offers 
static  pages  and  will  be  SSL  enabled  as  a  minimum. 

2.  Application  tier,  authenticates  authorized  users,  access,  and 
interfaces  between  the  user  and  the  data. 

3.  Protection  of  the  database  or  data  tier  (for  example,  flat 
files,  e-mail),  information  that  is  accessed  by  the  application 
on  behalf  of  the  user. 


The  COTS 
application  will 
use  only  approved 
access  controls 
between  the 
participant 
interface  and  the 
survey  database. 


NOTE:  DoDI  8500-2  and  8910-01  did  not  provide  relevant  guidance  beyond  AR  25-1  and  25-2.  Further,  specific 
regulations  concerning  COTS  are  to  be  provided  by  IMCEN.  These  regulations  are  currently  in  progress  and  so  will 
not  be  used  for  evaluation  of  a  COTS  product. 

IMPORTANT  CLARIFICATION:  In  the  previous  regulations,  it  is  assumed  that  the  term  “user”  will  refer  to  the 
ARI  developer  when  discussing  the  use  of  the  COTS  application  and  will  refer  to  the  survey  participant  when 
discussing  accessing  a  database  through  a  Web  interface. 
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Initial  COTS  Web-Based  Survey  Software  Product  List 
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Table  E-1 

Initial  COTS  Web-Based  Survey  Software  Product  List _ 

Vendor  Product  URL 


ActiveCampaign 

iSalient 

Anderson  Bell  Corp 

WebSurv 

Apian  Software 

SurveyPro 

Business  Objects,  an  SAP 
Company 

Business  Intelligence 

ChumpSoft 

perllQuestionnaire 

ChumpSoft 

phpQuestionnaire 

ClassApps 

SelectSurvey  ASP  ™ 

ClassApps 

SelectSurvey.NET 

Confirmit 

Confirmit  EEM 

Confirmit 

Confirmit  MR 

CustomerSat 

Self-Service 

Digvey 

Composer/  Eauncher/ 
Analyzer 

Electronic  Market 

Surveys 

QS  Survey  System 

Ennect  Online  Survey 
Software 

EnnectSurvey 

Golden  Hills  Software 

SurveyGold 

Grapevine  Surveys 

Grapevine 

Gravic,  Inc. 

Remark  Web  Survey 
Software 

Hosted  Survey 

Basic  ASP 

Hosted  Survey 

Enterprise  API 

iMagic  Software 

iMagic  Survey  Pro 

Infopoll 

Infopoll 

Inquisite 

Inguisite  EPM 

Inquisite 

Inquisite  Survey 

InSite  Survey  Systems, 

Inc. 

InSite  Survey  Systems 

InstantSurvey 

InstantSurvey 

Ipathia,  Inc 

SuperSurvey 

JetMan  Productions,  Inc. 

SurveyKey 

Jive  Software 

Clearspace  Community 

EiveSurveys 

EiveSurveys 

mySmartSolutions 

Surveyo 

ObjectPlanet,  Inc. 

Opinio 

PerfectEorm  (Eormerly 

PerfectEorms  (Eormerly 

Quask) 

Quask) 

PollCat 

Survey  Pro 

PollCat 

Surveys  Enterprise 

PollCat 

Surveys  for  Workgroups 

http://activecampaiqn.com/isalient/ 

http://www.andersonbeii.com/ 

http://apian.com/software/ 

http://www.businessobiects.com/?extcmp=07q3 

am  cio  tdwi  1 

http://www.chumpsoft.com/products/periq/ 

http://www.chumpsoft.com/products/phpq/ 

http://www.classapps.com/ 

http://www.ciassapps.com/ 

http://www.confirmit.com/ 

http://www.confirmit.com/ 

http://www.customersat.com/index2.htm 

http://www.diqivev.com/ 

http  ://www  .emssurvevs.com/ 

http://www.ennect.com/Survev/ 

http://www.survevqoid.com/ 

http://www.qrapevinesurvevs.com/ 

http://www.qravic.com/remark/websurvev/ 

http://www.hostedsurvev.com/home.htmi 

http://www.hostedsurvev.com/home.htmi 

http://www.imaqicsurvevsoftware.com/ 

http://infopoii.com/iive/survevs.dii/web 

http://www.inquisite.com/ 

http://www.inquisite.com/ 

http://www.insitesurvevs.com/Site/ 

http://www.instantsurvev.com/ 

http://www.supersurvev.com/ 

http://www.survevkev.com/ 

http://www.JiveSoftware.com/Communitv/ 

http://www.iivesurvevs.com/ 

http://www.survevo.com/ 

http://www.obiectpianet.com/opinio/ 

http://www.perfectfomis.com/ 

http://www.poiicat.com/ 

http://www.poiicat.com/ 

http://www.poiicat.com/ 
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Vendor  Product  _ URL _ 

http://www.UltimateSurvev.com/ 


Prezza  Technologies, 

Inc., 

Check  Box  Survey 

Pulse  Ware 

PulseWare 

Qualtrics 

SurveyZ! 

Questionmark 

Questionmark  Perception 

Raosoft 

Interform 

Reliant  Strategic  Talent 
Management  Solutions 

ZipSurvey 

Sawtooth  Technologies 

WebCanti  4.2  Mixed 

Inc  (used  by  the  Navy) 

Mode 

Snap  Surveys 

Snap  Professional 

Edition 

Space-Time  Research 

Self  Service  Business 
Intelligence 

SPSS,  Inc. 

Desktop  Author 

SPSS,  Inc. 

Dimensions 

SPSS,  Inc. 

Surveycraft 

Statpac,  Inc. 

Statpac  online  surveys 

SumQuest 

SumQuest 

Survey  Analystics 
Enterprise  Survey 
Software 

QuestionPro 

Survey  Grafter,  Inc. 

Survey  Crafter 
Professional 

Survey  System 

Enterprise  Edition 

Survey  System 

Professional  Edition 

SurveyMethods,  Inc. 

Advanced  Package 

SurveyMethods,  Inc. 

Professional  Package 

SurveyMonkey 

SurveyMonkey 

SurveySaid 

Basic  Researcher 

Package 

SurveySaid 

Internet  Researcher 
Package 

SurveySaid 

Total  Researcher 

Package 

SurveySite 

ComScore 

SurveyView 

SurveyView  Enterprise 
Web  Version 

Survey  Writer,  Inc. 

SurveyWriter 

Vision  Control 

Eusion 

Vision  Control 

Panel  -i- 

Vovici  Corp 

EEM  Communitv 

http://www.pulseware.com.au/ 

http://www.survevz.com/ 

http://www.questionmark.com/us/trvitout.asp 

X 

http://www.raosoft.com/products/interform/ 

http://www.zipsurvev.com/ 

http://www.sawtooth.com/ 

http://www.snapsurvevs.com/ 

http://www.str.com.au/ 

http://www.spss.com/dimensions/ 

http://www.spss.com/dimensions/ 

http://www.survevcraft.com/ 

http://www.statpac.com/ 

http://www.sumquest.com/ 

http://www.questionpro.com/ 

http://www.survevcrafter.com/interim2/defauit.as 

e 

http://www.survevsvstem.com/ 

http://www.survevsvstem.com/ 

http://www.survevmethods.com/ 

http://www.survevmethods.com/ 

http://www.survevmonkev.com/ 

http://www.survevsaid.com/ 

http://www.survevsaid.com/ 

http://www.survevsaid.com/ 

http://www.comscore.com/soiutions/survevsite.as 

http://www.survevview.com/ 

http://www.survevwriter.com/site/index.htmi 

http://www.VisionCriticai.com/Pro/ 

http://www.VisionCriticai.com/Pro/ 

http://www.vovici.com/ 
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Vendor 

Product 

URL 

Vovici  Corp 

EFM  Feedback 

httD://www.  vovici. com/ 

Webmyne  Systems  Inc. 

MagicSurveyTool 

httD://www.  maaicsurvevtooi.com/ 

Widgix  Software 

SurveyGizmo 

httD://www. survevaizmo.com/ 

WISCO  Computing 

WISCO  Survey  Power 

httD://www.wiscosurvev.com/ 

WorlApp 

Key  Survey 

httD://www. kevsurvev.com/ 

Xorbix  Technologies, 

Inc. 

loxphere 

httD://www.  ioxDhere.com/ 

ZapSurvey 

ZapSurvey 

httD://zaDSurvev.com/ 

Zarca  Interactive,  Inc. 

Zarca 

httD://www. zarca.com/ 

Zoomerang 

Zommerang  Pro 

httD://www. zoomerana.com/ 
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Appendix  F 

Market  Analysis 
Software  Testing  Protocol 
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Software  Testing  Protocol:  Steps  to  Test  COTS  Survey  Software  Product  Finalists 

Note:  Bulleted  numbers  refer  to  criteria  tested  in  each  step. 

1 .  Develop  sample  survey 


o 

2.03 

o 

2  17 

o 

3.12 

o 

2.08 

o 

2.19 

o 

3.13 

o 

2.09 

o 

2.29 

o 

3.15 

o 

2.10 

o 

3.01 

o 

3.16 

o 

2.11 

o 

3.03 

o 

3.24 

o 

2.12 

o 

3.05 

o 

3.27 

o 

2.13 

o 

3.06 

o 

3.33 

o 

2.14 

o 

3.07 

o 

4.01 

o 

2.15 

o 

3.08 

o 

2.16 

o 

3.09 

2.  Check  if  you  can  program  branching  by  writing  logic  statements  (as  opposed  to  using 
drop-downs) 

o  3.19 

3.  Edit  the  “Next”  navigation  button’s  label  to  say  “Next  Page” 

o  2.18 

4.  Try  to  label  the  first  page  as  Pageltest  (if  option  is  available). 

o  3.33 

5.  Set  up  the  survey  so  that  participants  can  review  their  answers  prior  to  submitting  the 
survey  (if  the  option  is  given  in  the  product). 

o  2.21 

o  3.36 

o  4.02 

6.  Set  up  the  survey  so  that  participants  can  withdraw  or  clear  all  their  survey  answers  (if 
the  option  is  given  in  the  product). 

o  2.31 

o  3.25 

7.  Set  up  the  survey  so  that  participants  can  clear  a  page  of  answers  (if  option  is  given  in 
product). 

8.  Try  to  print  out  the  question  properties  for  the  survey  (e.g.,  page,  question  variable  name, 
question  text,  etc.) 

o  2.22 

9.  Display  and  then  undisplay  question  numbers. 

o  3.04 
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10.  Spell  check  all  questions  and  instructions. 

o  2.20 
o  4.09 

11.  Set  up  the  export  variable  name  to  be  Qltest  and  Q2test  for  first  two  questions. 

12.  Set  up  the  export  response  option  names. 

13.  Save  2  questions  and  1  response  option  to  the  question  library. 

o  3.34 

14.  Export  survey  to  Word  document  (or  PDF)  if  this  is  the  only  option  and  rate  how  close  to 
the  online  survey  it  is. 

o  2.23 
o  3.20 
o  3.21 

15.  Publish  or  Activate  the  survey  (so  it  is  finalized  and  in  the  field). 

16.  Upload  participant  list  and  set  up  initial  email  to  distribute  survey  link. 

o  2.27 

17.  See  if  system  will  automatically  create  user  IDs  and  passwords  or  if  you  need  to  assign 
them. 

o  2.26 

18.  Send  link  through  mail  campaign  to  everyone’s  ICE  email 

o  4.12 

19.  Send  link  through  email  campaign  at  a  separate  time  to  participant’s  personal  emails  with 
a  Word  document  attachment. 

o  3.28 

20.  Follow-up  with  participants  in  both  batches  to  see  what  time  they  received  the  survey  link 
email. 

o  2.27 

21.  Take  Survey  lx  with  fake  data  (try  to  clear  an  answer  and  clear  a  page  of  data) 

o  2.07 

22.  Add  a  multiple  choice  question  after  Q1  asking  their  favorite  color  to  see  if  questions  can 
be  added  after  survey  is  launched. 

o  3.22 

23.  Ask  another  team  member  to  fill  out  first  2  pages  of  data. 
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24.  Export  to  SPSS 
o  2.02 


25.  Check  how  data  is  in  the  newly  added  q  (should  have  data  for  2"‘^  person,  but  no  data  for 
first  person). 

o  3.29 

26.  Check  tracking  status  of  participants. 

o  2.25 
o  3.23 

27.  Remove  an  email  from  list  so  they  do  not  receive  reminder  email. 

28.  Send  reminder  email  to  only  those  who  did  not  complete  the  survey  of  XX  rank 

o  2.04 


29.  Follow-up  with  person  whose  name  was  removed  to  ensure  they  did  not  receive 
reminder, 
o  2.24 


30.  Export  data  to  SPSS,  SAS,  &  Access  (if  allowed) 


o 

2.02 

o 

2.28 

o 

3.02 

o 

3.11 
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Appendix  G 

Market  Analysis 
Software  Testing  Questionnaire 
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ARI  Survey  Software  Test  Questionnaire 

<insert  header  picture> 


Pagel  Name  label 


Q.  Respondent  Info  and  Demographics  jlnfo  will  be  from  external  source) 

FName: _ 

LName: _ 

Phone: _ 

Email: _ 

ID: _ 

Age: _ 

Division: _ {branch  survey  based  on  division) 

Rank: _ 


DIV  A  (HR)  questions  {section  headingsj 
Q:  Are  you  familiar  with  survey  Web  applications? 

Yes  (skip  to  Web  app  question)  {write  code  for  sk^^ingj 
No 

Q:  How  many  job  openings  do  you  anticipate  having  within  the  next  year,  in  the  following 
divisions?  (Matrix  question-  input/drop  down) 


Division(s) 

#  of  openings 
(input/drop  down) 

DivA 

DivB 

DivC 

Q:  Which  of  the  following  topics  are  related  to  DIV  A?  (Mark  all  that  apply)(/?a«dQ/nfee 
Answer  choices  ri 

A 

B 

C 

D 


Other:  please  specify: 
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Q:  What  is  the  percent  o 


'  gross  income  is  derived  from  the  following  sectors  in  division  A? 


Division  A  sectors 

Percent  allocation 

A 

B 

C 

D 

Running  total 

Q.  WebApp(label):  Will  the  Weh  application  have  the  following  capabilities  below?  Large 
matrix  question  (To  find  out  formatting  capability  of  application  add  question  to  library) 


Topics 

Yes/No 

(Dropdown 

options) 

Level  of 
criticality 

(Dropdown 

options) 

Type  of 
Application 

(Dropdown 

options) 

Make  a  suggestion  for  one  or 
two  questions’  variabie  export 
names  and  iabeis  and 
response  option  vaiues. 

Yes/No 

Abiiity  to  skip/ask  particuiar 
questions  based  upon  record 
data/demographic  data  (i.e. 
piping  data  from  preioaded 
demographic  data  fiie) 

Yes/No 

Abiiity  to  have  piped  answers 
shown  to  ensure  participant 
shouid  be  exited  from  survey 
(e.g.,  you  seiected  you  were 
not  depioyed  over  the  past  12 
months.  Is  this  correct?) 

Yes/No 

Abiiity  to  show  previous 
answers  in  the  text  of  current 
questions  (i.e.  piping  into 
question  text) 

Yes/No 

Advanced  Formatting  (e.g., 
iine  spacing,  iine  indent, 
coiumn  iabeis,  iine  breaks, 
response  coiumn  widths, 
wrapping  in  coiumns) 

Yes/No 

DIV  C  (IT  Help)  questions 

Q:  Please  select  the  products  you  have  used  in  the  past  year  mark  all  that  apply  (Piping  In-TeM 
Question) 

Product  A 
Product  B 
Product  C 
Product  D 
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Q:  Please  rate  your  satisfaction  with  the  products  you  selected  in  the  previous  question 


Products 

Satisfaction 

<i>  (Piped  here)  </i> 
fhtml  question  taps) 

Very  Satisfied 

Somewhat 

Neutral 

Somewhat  Dissatisfied 

Very  Dissatisfied 

Product 

Rank  (1-4) 

Rate  (1-10) 

Product  A 

1-4 

1-10 

Product  B 

1-4 

1-10 

Product  C 

1-4 

1-10 

Product  D 

1-4 

1-10 

Q.  How  well  did  your  pre-deployment  training  prepare  you  to 
perform  full  spectrum  operations?  (definition  as  a  pop-up:  The  comjponents  of  full  spectrum 
wperations  are  1 )  Offense  2)  Defense  3)  Stab^^  4)  Civil  support) 

O  Very  well 
O  Well 

O  Neither  well  nor  poorly 
O  Poorly 
O  Very  poorly 

O  Not  applicable;  I  did  not  receive  full  spectrum  operations  training  prior  to  deploying. 
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Q.  This  is  a  matrix  question  involving  different  types  of  response  options. 


Topics 

Choice 

(Mark  all  that  apply) 

Choice 

(Mark  only  one) 

Choice 

(Input/comment) 

Choice 

(N/A  exclusivity?) 

A 

B 

C 

D 

E 

Q.  How  well  did  your  pre-deployment  training  prepare  you  to  perform  full  spectrum 
operations? 
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Appendix  H 
Market  Analysis 

COTS  Web-Based  Survey  Software  Review  Results 
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Table  H-1 

COTS  Web-Based  Survey  Software  Review:  Products  Eliminated  in  First  Round  (based  on 
“top few  eliminator”  criteria) 

Vendor 

Product 

Reason  Eliminated 

Anderson  Bell  Corp 

WebSurv 

■ 

Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

ChumpSoft 

perllQuestionnaire 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

ChumpSoft 

phpQuestionnaire 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Confirmit 

Confirmit  MR 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

CustomerSat 

Self-Service 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Electronic  Market 
Surveys 

QS  Survey  System 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Ennect  Online  Survey 
Software 

EnnectSurvey 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Golden  Hills  Software 

SurveyGold 

■ 

Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

Grapevine  Surveys 

Grapevine 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Gravic,  Inc. 

Remark  Web  Survey 
Software 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Hosted  Survey 

Basic  ASP 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Hosted  Survey 

Enterprise  API 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Infopoll 

Infopoll 

■ 

■ 

Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

Vendor  is  headquartered  outside  of  the 
United  States. 

Inquisite 

Inquisite  EPM 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

InSite  Survey  Systems, 
Inc. 

InSite  Survey  Systems 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

InstantSurvey 

InstantSurvey 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Ipathia,  Inc 

SuperSurvey 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 
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Vendor 

Product 

Reason  Eliminated 

JetMan  Productions,  Inc. 

SurveyKey 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Jive  Software 

Clearspace  Community 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

LiveSurveys 

LiveSurveys 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

mySmartSolutions 

Surveyo 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

ObjectPlanet,  Inc. 

Opinio 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

PollCat 

Survey  Pro 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

PollCat 

Surveys  Enterprise 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

■  Surveys  must  be  developed  and  hosted  on 

PollCat 

Surveys  for  Workgroups 

vendor’s  servers. 

PulseWare 

PulseWare 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Reliant  Strategic  Talent 

ZipSurvey 

■  Surveys  must  be  developed  and  hosted  on 

Management  Solutions 

vendor’s  servers. 

Sawtooth  Technologies 

WebCanti  4.2  Mixed 

■  Rating  of  vendor  reputation,  size  of 

Inc  (used  by  the  Navy) 

Mode 

company,  and  quality  indicated 
questionable  company  longevity. 

Space-Time  Research 

Self  Service  Business 

■  Surveys  must  be  developed  and  hosted  on 

Intelligence 

vendor’s  servers. 

SPSS,  Inc. 

Surveycraft 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Statpac,  Inc. 

Statpac  online  surveys 

■  Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

SumQuest 

SumQuest 

■  Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

Survey  Analystics 
Enterprise  Survey 
Software 

QuestionPro 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Survey  System 

Enterprise  Edition 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Survey  System 

Professional  Edition 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

SurveyMethods,  Inc. 

Advanced  Package 

■  Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 
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Vendor 

Product 

Reason  Eliminated 

SurveyMethods,  Inc. 

Professional  Package 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

SurveyMonkey 

SurveyMonkey 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

SurveySaid 

Basic  Researcher 

Package 

■ 

Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

SurveySaid 

Internet  Researcher 
Package 

■ 

Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

SurveySaid 

Total  Researcher 

Package 

■ 

Rating  of  vendor  reputation,  size  of 
company,  and  quality  indicated 
questionable  company  longevity. 

SurveySite 

ComScore 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

SurveyView 

SurveyView  Enterprise 
Web  Version 

■ 

Vendor  is  headquartered  outside  of  the 
United  States. 

Survey  Writer,  Inc. 

SurveyWriter 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Vision  Control 

Fusion 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Vision  Control 

Panel  -i- 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Webmyne  Systems  Inc. 

MagicSurveyTool 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Widgix  Software 

SurveyGizmo 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

WISCO  Computing 

WISCO  Survey  Power 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Xorbix  Technologies, 

Inc. 

loxphere 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

ZapSurvey 

ZapSurvey 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Zoomerang 

Zommerang  Pro 

■ 

Surveys  must  be  developed  and  hosted  on 
vendor’s  servers. 

Note.  Products  passed  this  round  of  review  if  the  vendor  reported  that  theirl)  survey  development  software  could  be 


installed  on  ARl  servers;  2)  survey  hosting  software  and  software  databases  could  be  installed  on  ARI  servers;  3) 
hosting  software  allowed  individuals  to  enter  survey  questions  and  manipulate  survey  format  through  a  graphical 
user  interface  (GUI)  (i.e.,  non-coding  interface);  4)  company  reputation,  size,  and  quality  combined  was  rated  a  1= 
solid,  or  2  =  ok;  and  5)  company  was  headquartered  within  the  United  States. 
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Table  H-2 


COTS  Web-Based  Survey  Software  Review:  Products  Eliminated  in  Second  Round  (based  on 
“critical”  criteria) _ 


Vendor 

Product 

Reason  Eliminated 

ActiveCampaign 

iSalient 

■  Product  does  not  allow  for  58%  of 
“critical”  features,  including... 

o  Product  does  not  allow  data  to  be 
exported  to  a  format  compatible  with 
SPSS  or  SAS. 

o  Product  does  not  have  the  ability  to 
do  advanced  branching  forward  in  the 
survey. 

o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 

Apian  Software 

SurveyPro 

■  Product  does  not  allow  for  19%  of 
“critical”  features,  including... 
o  Product  does  not  have  the  ability  to 
do  advanced  branching  forward  in  the 
survey. 

o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 

Business  Objects,  an  SAP 
Company 

Business  Intelligence 

■  Vendor  does  not  offer  phone  support  and 
has  not  responded  via  email. 

ClassApps 

SelectSurvey  ASP  ™ 

■  Product  does  not  allow  for  16%  of 
“critical”  features,  including... 
o  Product  is  not  as  advanced  as  other 
programs. 

ClassApps 

SelectSurvey.NET 

■  Product  does  not  allow  for  16%  of 
“critical”  features,  including... 
o  Product  is  not  as  advanced  as  other 
programs. 

Digvey 

Digivey  + 

■  Product  does  not  allow  for  26%  of 
“critical”  features,  including... 
o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 
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Vendor 

Product 

Reason  Eliminated 

Electronic  Market 
Surveys 

QS  Survey  System 

■  Vendor  only  offers  full-service  survey 
process  and  does  not  allow  for  clients  to 
develop  surveys,  analyze  data,  etc. 

iMagic  Software 

iMagic  Survey  Pro 

■  Vendor  does  not  offer  phone  support  and 
has  not  responded  via  email. 

Prezza  Technologies, 
Inc., 

Check  Box  Survey 

■  Product  does  not  allow  for  29%  of 
“critical”  features. 

Questionmark 

Questionmark  Perception 

■  Product  does  not  allow  for  26%  of 
“critical”  features,  including... 
o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 

SPSS,  Inc. 

Desktop  Author 

■  Product  does  not  allow  for  32%  of 
“critical”  features,  including... 
o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 

Survey  Grafter,  Inc. 

Survey  Grafter 
Professional 

■  Product  does  not  allow  for  16%  of 
“critical”  features,  including... 
o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 

Vovici  Corp 

EFM  Feedback 

■  Product  does  not  allow  for  19%  of 
“critical”  features. 

Note.  Products  passed  this  round  of  review  if  the  vendor  reported  that  greater  than  85%  of  the  “critical”  criteria  and 
all  of  the  absolutely  essential  features  were  available. 
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Table  H-3 


COTS  Web-Based  Survey  Software  Review:  Products  Eliminated  in  Third  Round  (based  on 
“improves  greatly”  criteria) _ 


Vendor 

Product 

Reason  Eliminated 

Inquisite 

Inquisite  Survey 

■  Product  does  not  allow  for  19%  of 
features  that  would  greatly  improve 
survey  development  and  administration. 

PerfectForm  (Formerly 

PerfectForms  (Formerly 

Quask) 

Quask) 

■  Product  does  not  allow  for  17%  of 
features  that  would  greatly  improve 
survey  development  and  administration. 

Qualtrics 

SurveyZ! 

■  Product  does  not  allow  for  1 1%  of 
features  that  would  greatly  improve 
survey  development  and  administration, 
including. . . 

o  Company  does  not  typically  allow  for 
development  and  hosting  on  client 
servers  (though  can  allow  for  it). 

■  Less  positive  reputation,  size  of  company, 
or  longevity  of  company  than  other 
products. 

Snap  Surveys 

Snap  Professional 

■  Product  does  not  allow  for  53%  of 

Edition 

features  that  would  greatly  improve 
survey  development  and  administration, 
including. . . 

o  Product  does  not  have  the  ability  to 
filter  reminder  email  recipients  (e.g., 
send  to  specific  sub-samples  based  on 
rank). 

SPSS,  Inc. 

Dimensions 

■  Product  portion  with  easy  user  interface 
has  restrictive  item  limits  (e.g.,  100  pieces 
of  information). 

■  Product  portion  with  unlimited  items 
allowed  does  not  have  easy  graphical  user 
interface  (i.e.,  requires  coding). 

Note.  Products  passed  this  round  of  review  if  the  vendor  reported  that  greater  than  90%  of  the  “improves  greatly” 
criteria  were  available. 
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Table  H-4 


COTS  Web-Based  Survey  Software:  Products  Advanced  to  Final  Testing  Round  * 


Vendor 

Product 

URL 

Confirmit 

Confirmit  EFM 

httn  ://www  .confirmit.com/ 

Vovici  Corp 

EFM  Community 

httD://www. vovici.com/ 

WorlApp 

Key  Survey 

httD://www.kevsurvev.com/ 

Zarca  Interactive,  Inc. 

Zarca 

httm/Zwww. zarca.com/ 

*See  Appendix  J  for  a  detailed  breakdown  of  testing  results. 
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Appendix  I 
Market  Analysis 

Detailed  Functions  and  Features  Review  of  Prezza  Technologies  Checkbox  Survey, 
Question  Mark  Perception,  SPSS  Desktop  Author,  and  SPSS  Dimensions 
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Table  I-l 


Detailed  Functions  and  Features  Review  of  Prezza  Technologies  Checkbox  Survey,  Question  Mark  Perception, 
SPSS  Desktop  Author,  and  SPSS  Dimensions _ 


Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

“CRITICAL”  FEATURES 

2.01 

Meets  Army  and  DoD  security 
regulations 

X 

X 

2.02 

Ability  to  export  data  to  a  format 
compatible  with  SPSS  or  SAS 

Feature  is  available 
via  Excel  or  ASCII  file 
format. 

2.03 

Ability  to  do  advanced 
branching  forward  in  the  survey 

2.04 

Ability  to  filter  reminder  email 
recipients  (e.g.,  send  to  specific 
sub-samples  based  on  rank) 

X 

X 

2.05 

Unlimited  number  of  responses 
to  surveys 

X 

2.06 

Thorough  and  clear 
documentation/training 

2.07 

Ability  to  create  professional¬ 
looking  surveys 

2.08 

User-friendly  developer 
interface,  including  ease  of 
formatting  question  and  response 
option  text  (e.g.,  bold,  italics) 

y 

y 

2.09 

Ability  to  skip/ask  particular 
questions  based  upon  record 
data/demographic  data  (i.e. 
piping  data  from  preloaded 
demographic  data  file) 

y 

2.10 

Ability  to  show  previous 
answers  in  the  text  of  current 
questions  (i.e.  piping  into 
question  text) 

X 

2.11 

Ability  to  add  HTML  code  to 
question  text  and/or  question 
responses 

^  xt 

^  Html  can  be  added 
within  question  text, 
but  not  question 
responses. 
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Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

2.12 

Advanced  formatting  (e.g.,  line 
spacing,  line  indent,  column 
labels,  line  breaks,  response 
column  widths,  wrapping  in 
columns) 

2.13 

Ability  to  write/code  survey 
branching  logic  (rather  than  only 
having  the  option  to  use  a  drop 
down  list) 

X 

X 

2.14 

Ability  to  name  (label)  questions 

2.15 

Ability  to  add  comment  boxes, 
so  participants  can  add 
comments  throughout  the  survey 

2.16 

Ability  to  add  graphics  in  the 
header  (at  beginning  of  survey) 

2.11 

Ability  to  add  graphics  in  the 
question  text 

y 

y 

y 

2.18 

Ability  to  edit  navigation 
button's  label  (e.g.,  help  button, 
FAQ  button,  privacy  act  button, 
submit  button) 

2.19 

Customizable  templates  (e.g., 
theme  or  cascading  style  sheets) 

^  Feature  is 
accomplished  via  the 
use  of  the  survey's 
template  controlled  by 
style  sheets. 

2.20 

Spell  Check 

X 

2.21 

Offer  respondents  an  easy  way 
to  review  their  answers  to 
previous  questions  (i.e.  without 
having  to  backtrack  through 
every  question)  (e.g.,  jump 
function) 

X 

X 

^  The  review  happens 
through  use  of  a  jump 
feature  that  is 
embedded  in  some 
templates. 

2.22 

Ability  to  print  out  the  question 
properties  for  the  survey  (e.g., 
page,  question  variable  name, 
question  text,  and  question  scale) 

X 

X 

X 

^  The  authoring  tool 
allows  you  to  print  out 
the  survey. 

2.23 

Ability  to  export  survey  question 
text  and  response  options  to 
external  document  (e.g.,  MS 
Word) 

X 
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Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

2.24 

Ability  to  remove  a  respondent 
from  the  survey  distribution  list 
(during  field  period)  so  they  do 
not  receive  future 
correspondence  (e.g.,  reminder 
email) 

X 

2.25 

Ability  to  track  the  exact 
question  the  participant 
answered  prior  to  abandoning 
the  survey 

X 

x+ 

X 

^  The  program  allows 
for  survey  developers 
to  access  the  raw  data 
within  the  program, 
where  you  can  check 
the  point  at  which  each 
participant  abandoned 
the  survey,  but  it  does 
not  display  in  the 
tracking  area. 

2.26 

Ability  to  automatically  create 
and  assign  IDs/passwords 

x+ 

X 

^  Feature  is 
unavailable,  but  could 
do  it  with  additional 
consulting  services. 

2.21 

Ability  to  filter  email  recipients 
(e.g.,  send  only  to  participants 
who  have  not  completed  the 
survey,  send  initial  email  based 
on  a  participant  database 
variable  to  overcome  limits  on 
sending  mass  emails) 

X 

X 

X 

2.28 

Ability  to  export  variable  names 
and  labels  (with  data)  for  use  in 
SPSS  or  SAS 

X 

X 

2.29 

Ability  to  assign  variable  labels 
within  the  application 

X 

2.30 

Ability  to  assign  export  response 
option  values 

2.31 

Ability  for  participant  to 
withdraw  a  survey  (either  one 
that  has  been  electronically 
submitted  or  one  that  is  partially 
complete) 

X 

X 

X 

X 

Percent  of  “Critical”  Features  Available 

67.74% 

70.97% 

67.74% 

93.55% 

1-4 


Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

“IMPROVES  GREATLY”  FEATURES 

3.01 

Ability  to  assign  export  variable 
names 

3.02 

Ability  to  export  data  directly  to 
SAS 

X 

X 

X 

3.03 

Ability  to  have  different  types  of 
response  options  in  the  same 
matrix  question 

y 

3.04 

Automatic  numbering  with  the 
option  of  displaying  numbers  or 
not 

x+ 

^  Plans  include  feature 
to  be  added  in  2009. 

3.05 

The  ability  to  add  to  and  select 
from  a  response  option  library  of 
commonly-used  response  option 
sets 

X 

x+ 

^  Plans  include  feature 
to  be  added  in  2009. 

3.06 

The  ability  to  add  to  and  select 
from  a  question  library  of  full 
questions  (question  text  and 
response  options) 

3.07 

Ability  to  show  only  the 
response  options  that  were 
selected  (or  not)  in  a  previous 
question  (i.e.  piping  into 
response  options) 

X 

X 

3.08 

Ability  to  randomize  the  order  of 
response  options 

y 

3.09 

Automatically  add  response 
values  as  each  response  is  added 
(e.g.,  summing  percents) 

X 

x+ 

^  Plans  include  feature 
to  be  added  in  2009. 

3.10 

System  keeps  historical  log  files 
(i.e.,  that  record  upload,  deletion, 
and  emails  to  survey 
respondents)  that  cannot  be 
deleted  by  ARI 

X 

3.11 

Ability  to  export  data  directly  to 
Microsoft  Access 

X 

3.12 

Ability  to  randomize  the  order  of 
questions 

^  ”  Feature  is  available 
via  scripting  snippets. 
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Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

3.13 

Ability  to  randomize  the  order  of 
pages 

X 

^  ”  Feature  is  available 
via  scripting  snippets. 

3.14 

Qualitative  data  analysis 
functionality  (e.g.,  keying  in  on 
words  or  phases) 

X 

x+ 

x^ 

X  " 

^  Plans  include  feature 
to  be  added  in  2009. 

^  ”  Feature  is  only 
available  through 
coding  in  a  mix  of 
HTML,  XML,  and 

CSS. 

3.15 

Ability  to  supplement  easy 
developer  interface  with  HTML 
or  other  coding  for  more  control 

X 

x+ 

^  Plans  include  feature 
to  be  added  in  2009. 

3.16 

Ability  to  create  online  help 
features  (e.g.,  pop-up  boxes  or  a 
mouse  roll-over  function)  that 
displays  definitions 

X 

x+ 

^  Plans  include  feature 
to  be  added  in  2009. 

3.17 

Ability  to  import  survey 
questions  from  external 
application,  such  as  MS  Word 

X 

y 

y 

y 

3.18 

Ability  to  branch  participants 
backwards  in  the  survey 

X 

y 

y 

3.20 

Ability  to  export  identical  survey 
(i.e.,  formatting  included)  to 
external  document  (e.g.,  MS 
Word) 

X 

x^ 

X  " 

^  ”  The  questions  are 
identical,  but 
formatting  is  not 
included. 

3.21 

Ability  to  add  comments  to  the 
survey  once  it  is  exported 

X 

x+ 

X 

^  Plans  include  feature 
to  be  added  in  2009. 

3.22 

Ability  to  change  survey 
questions  or  response  options 
after  survey  is  in  field  without 
corrupting  the  data 

X 

X 

3.24 

Ability  to  do  ranking  and  rating 
questions  in  one  question  type 

y 

y 

3.25 

Respondents  ability  to  withdraw 
a  survey  (e.g.,  fill  out  paper  form 
and  decide  not  to  submit) 

X 

X 

^  Feature  requires  Print 
(&  Scan  add-on. 

3.26 

Ability  to  do  sampling  for 
extremely  long  surveys,  so  each 
participant  does  not  have  to 
complete  entire  survey 

X 

X 
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Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

3.27 

Ability  to  have  different 
response  options  within  one 
matrix  question  (e.g.,  sometimes 
N/A  and  other  times  no  N/A) 

X 

x+ 

^  Plans  include  feature 
to  be  added  in  2009. 

3.28 

Ability  to  attach  external 
documents  to  the  dissemination 
email  (e.g.,  a  support  letter  from 
high  in  DoD)  or  ability  to  imbed 
email  signatures  into  the 
dissemination  email 

X 

x+ 

X 

^  Plans  include  feature 
to  be  added  in  2009. 
Requires  additional 
consulting  services. 

3.29 

Ability  for  participant  to  clear  a 
page  or  a  specific  question  (e.g., 
radio  button) 

X 

x+ 

X 

^  Plans  include  feature 
to  be  added  in  2009. 

3.30 

Ability  to  export  data  directly  to 
SPSS  or  SAS 

X 

X 

^  Data  can  be  exported 
via  ASCII  or  EXCEL. 

3.31 

System  automatically  creates  an 
archive/back-up  of 
questionnaire/data 

X 

y 

¥  <T 

The  application  has 
a  backup  utility  that 
stores  backup  on  the 
same  machine.  If  you 
want  to  backup  to  a 
different  location,  you 
have  to  set  that  up 
outside  the  application. 

3.32 

Advanced  Page  Layout  (e.g., 
resize  objects,  layer  objects, 
snap  objects  to  a  grid,  and  lock  a 
page) 

X 

X 

^  ”  Eeature  is  only 
available  through 
coding  in  a  mix  of 
HTML,  XML,  and 

CSS. 

3.33 

Ability  to  name  (label)  pages 

X 

X 

3.35 

Administrative  control,  such  as 
being  able  to  remove  cases  or 
reset  cases  (rather  than  going 
into  IMCEN  to  do  so)  to  allow 
participants  who  were 
inappropriately  exited  from  the 
survey  back  in 

X 

X 

Percent  of  “Improves  Greatly”  Features 
Available 

25.00% 

50.00% 

68.75% 

93.75% 
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Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

“IMPROVES  SOMEWHAT”  FEATURES 

4.01 

Ability  to  add  graphics  in 
response  options 

V 

V 

V 

4.03 

Option  of  exporting  only  list  of 
questions  VS  entire  survey 

X 

4.04 

Optical  scanning  features 

X 

X 

Feature  is 

accomplished  via  Print 
&  Scan  add-on. 

4.05 

Ability  for  platform  to 
automatically  dump  test  data 
once  activated 

X 

X 

X 

4.06 

Feature  that  allow  developers  to 
easily  check  if  skip  patterns 
work  without  taking  entire 
survey  as  participant 

X 

X 

X 

4.07 

Ability  to  do  data  cleaning  and 
some  data  analysis  in  the 
application 

X 

^  This  can  only  be  done 
within  one  of  12 
canned  reports,  which 
is  set  to  do  some 
analysis  for  you.  Any 
additional  analyses  you 
want  must  be  done 
through  a  customized 
report  that  would 
require  additional 
consulting  services. 

4.08 

Ability  to  display  basic 
figures/tables  (e.g.,  cross  tab)  of 
certain  important  questions  (e.g., 
ability  to  look  at  non-responses 
by  rank) 

X 

4.09 

Spell  check  library  where 
commonly-used  words  or 
acronyms  can  be  added 

X 

X 

X 

4.10 

Meta-data  capabilities 

X 

4.11 

Ability  to  store  the  data  in  an 
encrypted  format 

X 

X 

x^ 

x" 

^  ”  Feature  is  not 
officially  supported 
until  version  5.6. 
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Criteria 

# 

Criteria  Description 

Prezza 
Technologies 
Check  Box 
Survey 

Question 

mark 

Perception 

SPSS 

Desktop 

Author 

SPSS 

Dimensions* 

Comments 

4.12 

Ability  to  create  professional¬ 
looking  emails 

X 

4.13 

Publish  a  survey  to  a 
transportable  file  that  can  be  run 
on  local  machines  (e.g.,  non- 
Internet) 

X 

^  Feature  is  available 
via  Questionmark  To 

Go  add-on. 

Percent  of  “Improves  Somewhat” 
Features  Available 

33.00% 

66.66% 

50.00% 

75.00% 

Total  Percent  of  Features  Available 

44.00% 

12M% 

65.53% 

93.33% 

Feature  available;  Feature  not  available. 

*  SPSS  Dimensions  has  a  development  limitation  of  100  pieces  of  information  (i.e.,  surveys  with  more  than  100  pieces  of  information, 
including  questions  and  instructions,  can  only  be  developed  through  coding).  This  limitation  caused  us  to  eliminate  SPSS  dimensions  from 
further  review.  Note:  Criteria  3.19,  3.23,  3.34,  3.36,  and  4.02  were  eliminated  because  of  duplication  with  other  criteria  listed. 
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Appendix  J 
Market  Analysis 

Detailed  Functions  and  Features  Review  of  COTS  Web-based  Survey  Software  Finalists 
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Table  J-1 

Detailed  Functions  and  Features  Review  of  COTS  Web-based  Survey  Software  Finalists 


Criteria 

# 

Criteria  Description 

Confirmit 

EEM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

Comments 

“CRITICAL”  FEATURES 

2.01 

Meets  Army  and  DoD  security 

regulations 

(NA) 

(NA) 

(NA) 

(NA) 

2.02 

Ability  to  export  data  to  a 

§ 

'  Data  can  be  exported 

format  compatible  with  SPSS 

(5) 

(5) 

(5) 

(5) 

to  MS  Excel,  CSV, 

or  SAS 

SPSS,  XML,  or  MS 
Access  format. 

2.03 

Ability  to  do  advanced 

branching  forward  in  the  survey 

(5) 

(5) 

(5) 

(2.5) 

2.04 

Ability  to  filter  reminder  email 

recipients  (e.g.,  send  to 
specific  sub-samples  based  on 
rank) 

(5) 

(3.5) 

(2.5) 

(5) 

2.05 

Unlimited  number  of  responses 

^  There  is  a  database 

to  surveys 

(NA) 

(NA) 

(NA) 

(NA) 

limit  of  approximately 
4,000  variables. 

2.06 

Thorough  and  clear 

documentation/training 

(4) 

(4) 

(4.5) 

(2.5) 

2.07 

Ability  to  create  professional¬ 

looking  surveys 

(5) 

(4.5) 

(5) 

(3) 

2.08 

User-friendly  developer 

interface,  including  ease  of 
formatting  question  and 
response  option  text  (e.g.,  bold, 
italics) 

(4.5) 

(5) 

(1.5) 

(4) 

2.09 

Ability  to  skip/ask  particular 

questions  based  upon  record 
data/demographic  data  (i.e. 
piping  data  from  preloaded 
demographic  data  file) 

(5) 

(5) 

(5) 

(3.5) 

2.10 

Ability  to  show  previous 

answers  in  the  text  of  current 
questions  (i.e.  piping  into 
question  text) 

(4) 

(5) 

(1) 

(2.5) 

2.11 

Ability  to  add  HTML  code  to 

question  text  and/or  question 
responses 

(5) 

(5) 

(4.5) 

(5) 
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Criteria 

# 

Criteria  Description 

Confirmit 

EFM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

Comments 

2.12 

Advanced  formatting  (e.g.,  line 

spacing,  line  indent,  column 
labels,  line  breaks,  response 
column  widths,  wrapping  in 
columns) 

(5) 

(4) 

(2) 

(4) 

2.13 

Ability  to  write/code  survey 

§ 

'  Feature  is  not  yet 

branching  logic  (rather  than 

(3) 

(NA) 

(3.5) 

(NA) 

available,  but  can  be 

only  having  the  option  to  use  a 
drop  down  list) 

built  in  60  days. 

2.14 

Ability  to  name  (label) 

X 

§ 

'  Feature  will  be 

questions 

(5) 

(NA) 

(5) 

(5) 

available  in  the  next 
release  on  12/15/08. 

2.15 

Ability  to  add  comment  boxes. 

so  participants  can  add 
comments  throughout  the 
survey 

(5) 

(5) 

(4.5) 

(5) 

2.16 

Ability  to  add  graphics  in  the 

header  (at  beginning  of  survey) 

(5) 

(5) 

(5) 

(5) 

2.17 

Ability  to  add  graphics  in  the 

question  text 

(5) 

(4.5) 

(2) 

(2) 

2.18 

Ability  to  edit  navigation 

§ 

'  Feature  is  not  yet 

button's  label  (e.g.,  help  button, 
FAQ  button,  privacy  act  button, 
submit  button) 

(5) 

(5) 

(5) 

(NA) 

available,  but  can  be 
built  if  requested. 

2.19 

Customizable  templates  (e.g.. 

theme  or  cascading  style 
sheets) 

(4) 

(5) 

(4.5) 

(2.5) 

2.20 

Spell  Check 

”  Spell  check  feature 

(5) 

(5) 

(2) 

(5) 

exists  on  a  question  by 
question  basis  only. 

2.21 

Offer  respondents  an  easy  way 

5 

'  Feature  is  not  yet 

to  review  their  answers  to 

(2.5) 

(4) 

(5) 

(NA) 

available,  but  a  link  to 

previous  questions  (i.e.  without 

a  PDF  document  that 

having  to  backtrack  through 

displays  answers  for 

every  question)  (e.g.,  jump 

review  can  be  built  in 

function) 

30  days. 

111 

Ability  to  print  out  the  question 

X 

§ 

'  Feature  is  not  yet 

properties  for  the  survey  (e.g., 
page,  question  variable  name, 
question  text,  and  question 
scale) 

(5) 

(NA) 

(3.5) 

(NA) 

available,  but  can  be 
built  in  30  days. 
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Criteria 

# 

Criteria  Description 

Confirmit 

EFM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

2.23 

Ability  to  export  survey 

question  text  and  response 
options  to  external  document 
(e.g.,  MS  Word) 

(5) 

(5) 

(5) 

(5) 

2.24 

Ability  to  remove  a  respondent 

from  the  survey  distribution  list 
(during  field  period)  so  they  do 
not  receive  future 
correspondence  (e.g.,  reminder 
email) 

(5) 

(5) 

(5) 

(5) 

2.25 

Ability  to  track  the  exact 

question  the  participant 
answered  prior  to  abandoning 
the  survey 

(NA) 

(4) 

(2) 

(NA) 

2.26  Ability  to  automatically  create 

X 

x^ 

X 

and  assign  IDs/passwords 

(NA) 

(NA) 

(NA) 

(4.5) 

2.27 

Ability  to  filter  email  recipients 

(e.g.,  send  only  to  participants 
who  have  not  completed  the 
survey,  send  initial  email  based 
on  a  participant  database 
variable  to  overcome  limits  on 
sending  mass  emails) 

(5) 

(5) 

(2.5) 

(NA) 

2.28 

Ability  to  export  variable 

•/  x^ 

names  and  labels  (with  data)  for 
use  in  SPSS  or  SAS 

(3) 

(4) 

(5) 

(4.5) 

Comments 


Survey  exports  to 
PDF  format  only. 


This  feature  can  only 
be  performed  through 
exporting  the  data. 
There  is  a  filter  that 
allows  survey 
developers  to  export 
only  incomplete  data. 

5 

'  Feature  will  be 
available  in  the  next 
release  on  12/15/08. 

¥ 

This  product 
automatically  provides 
participants  with  a 
unique  link,  serving  a 
similar  purpose. 

5 

'  System  automatically 
creates  user  IDs  and 
passwords,  but 
manually  assigning 
IDs  and  passwords 
require  technical 
support. 

5 

'  Feature  is  not  yet 
available,  but  can  be 
built  in  60  days.  Can 
currently  select 
manually  which  email 
addresses  to  send 
emails. 

^  Export  directly  to 
SPSS  will  be  available 
in  the  next  release  on 
12/15/08;  Export  to 
SAS  is  expected  to  be 
added  in  2009. 


Cri^ria  Criteria  Description 

Confirmit 

EFM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

Comments 

2.29  Ability  to  assign  variable  labels 

X 

X 

within  the  application 

(NA) 

(NA) 

(3.5) 

NA 

2.30  Ability  to  assign  export 

response  option  values 

(5) 

(5) 

(5) 

(5) 

2.3 1  Ability  for  participant  to 

X 

X 

X 

§ 

'  Feature  is  not  yet 

withdraw  a  survey  (either  one 
that  has  been  electronically 
submitted  or  one  that  is 
partially  complete) 

(NA) 

(NA) 

(NA) 

(NA) 

available,  but  can  be 
built  in  60  days. 

Percent  of  “Critical”  Features 
Available 

90.32% 

93.55% 

74.19% 

“Critical”  Feature  Average  Usability 
Rating 

4.60 

4.67 

3.85 

4.07 

“IMPROVES  GREATLY”  FEATURES 

3.01  Ability  to  assign  export 

X 

X 

'  Feature  will  be 

variable  names 

(5) 

(NA) 

(NA) 

(5) 

available  in  the  next 
release  on  12/15/08. 

3.02  Ability  to  export  data  directly 

X 

X  ® 

X 

”  Data  can  be  exported 

to  SAS 

(NA) 

(NA) 

(NA) 

(NA) 

as  a  CSV,  MS  Excel, 

SPSS,  or  XML  format. 
^  The  ability  to  export 
directly  SAS  is 
expected  in  January 
2009. 


3.03 

Ability  to  have  different  types 

of  response  options  in  the  same 
matrix  question 

(3.5) 

(4) 

(2) 

(4.5) 

3.04 

Automatic  numbering  with  the 

X 

X 

option  of  displaying  numbers  or 

(NA) 

(NA) 

(5) 

(5) 

not 

3.05 

The  ability  to  add  to  and  select 

from  a  response  option  library 
of  commonly-used  response 
option  sets 

(3.5) 

(5) 

(5) 

(5) 

3.06 

The  ability  to  add  to  and  select 

s 

'  Feature  is  not  yet 

from  a  question  library  of  full 
questions  (question  text  and 
response  options) 

(4) 

(5) 

(5) 

(NA) 

available,  but  can  be 
built  in  60  days. 

Criteria 

# 

Criteria  Description 

Confirmit 

EFM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

Comments 

3.07 

Ability  to  show  only  the 

'  Feature  is  not  yet 

response  options  that  were 

(4) 

(4) 

(3.5) 

(NA) 

available,  but  can  be 

selected  (or  not)  in  a  previous 

built  in  60  days.  This 

question  (i.e.  piping  into 

feature  would  show  all 

response  options) 

answers  from  previous 
questions,  but  grey  out 
non-relevant  answers. 

3.08 

Ability  to  randomize  the  order 

of  response  options 

4.5 

(5) 

(5) 

(5) 

3.09 

Automatically  add  response 

values  as  each  response  is 
added  (e.g.,  summing  percents) 

(5) 

(5) 

(5) 

(5) 

3.10 

System  keeps  historical  log 

files  (i.e.,  that  record  upload, 
deletion,  and  emails  to  survey 
respondents)  that  cannot  be 
deleted  by  ARI 

(NA) 

(NA) 

(NA) 

(NA) 

3.11 

Ability  to  export  data  directly 

X 

X 

X 

to  Microsoft  Access 

(NA) 

(NA) 

(NA) 

(5) 

3.12 

Ability  to  randomize  the  order 

X 

5 

'  Feature  will  be 

of  questions 

(5) 

(NA) 

(4) 

(5) 

available  in  the  next 
release  on  12/15/08. 

3.13 

Ability  to  randomize  the  order 

❖ 

¥ 

Feature  will  be 

of  pages 

(4.5) 

(NA) 

(4.5) 

(NA) 

available  in  the  next 
release  on  12/2008. 

3.14 

Qualitative  data  analysis 

X  ® 

X 

”  Filters  that  sort  open- 

functionality  (e.g.,  keying  in  on 

(NA) 

(NA) 

(NA) 

(NA) 

ended  comments  based 

words  or  phases)  on  a  word  or  set  of 

words  are  available. 


3.15  Ability  to  supplement  easy 

developer  interface  with  HTML 
or  other  coding  for  more 
control 

(5) 

(4) 

(4.5) 

(5) 

3.16  Ability  to  create  online  help 

¥ 

Feature  will  be 

features  (e.g.,  pop-up  boxes  or 
a  mouse  roll-over  function)  that 
displays  definitions 

(3.5) 

(NA) 

(3.5) 

(NA) 

available  in  the  next 
release  on  12/2008. 

”  Can  create  a  question 
mark  icon  which 
displays  a  mouse-over 

pop-up  window, 
s 

'  Feature  is  not  yet 
available,  but  can  be 
built  in  60  days. 

-6 


Criteria 

# 


Criteria  Description 


Confirmit  Vovici  EFM  WorldApp  Zarca 
EFM  Community  Key  Survey  Interactive* 


Comments 


Ability  to  import  survey 
questions  from  external 
application,  such  as  MS  Word 


Plans  include  feature 
to  be  added  in  2009. 

5 

'  Feature  is  not  yet 
available,  but  can  be 
built  in  90  days. 


Ability  to  branch  participants 

X 

X 

X 

xS 

5 

'  Feature  is  not 

backwards  in  the  survey 

(NA) 

(NA) 

(NA) 

(NA) 

incorporated,  but  the 
restriction  can 
potentially  be  lifted  to 
allow  this. 

Ability  to  export  identical 

X 

x^ 

xS 

^  The  product  exports 

survey  (i.e.,  formatting 
included)  to  external  document 
(e.g.,  MS  Word) 

(NA) 

(NA) 

(5) 

(NA) 

the  survey  with 
identical  questions,  but 
the  formatting  does  not 

export. 

”  Feature  exports  to 

PDF  only, 
s 

'  Ability  to  print  the 
survey  and  all  pages  at 
a  time  (includes 
formatting)  but  cannot 
export  to  Word  or 
PDF.  There  is  an 
option  to  allow 
participants  to  print  the 
survey  as  well. 


Ability  to  add  comments  to  the 

X 

survey  once  it  is  exported 

(5) 

(5) 

(2.5) 

(NA) 

3.22  Ability  to  change  survey  •/  V  X*  ^  Deleting  or  changing 

questions  or  response  options  (5)  (3.5)  (3.5)  (NA)  questions  is  done 

after  survey  is  in  field  without  through  hiding  old 

corrupting  the  data  questions  and 

potentially  creating  a 
new  question.  All 
features  are  able  to  be 
changed  through 
HTML,  but  this  is  not 
recommended. 

5 

'  Feature  allows  fixing 
of  typos,  but  not 
adding  of  questions. 


Criteria 

# 


Criteria  Description 


Confirmit  Vovici  EFM  WorldApp  Zarca 
EFM  Community  Key  Survey  Interactive* 


Comments 


3.24  Ability  to  do  ranking  and  rating 
questions  in  one  question  type 


The  ranking  question 
format  takes  on  the 
form  of  3  columns 
with  all  the  responses 
and  the  participant 
chooses  one  response 
from  each  column. 

§ 

'  Feature  is  not  yet 
available,  but  can  be 
built  fairly  easily. 


3.25  Respondents  ability  to 

withdraw  a  survey  (e.g.,  fill  out 
paper  form  and  decide  not  to 
submit) 


Although  a 
respondent  is  not  able 
to  withdraw  a  survey, 
one  is  able  to  clear 
their  responses  on  a 
specific  page. 


Ability  to  do  sampling  for 
extremely  long  surveys,  so  each 
participant  does  not  have  to 
complete  entire  survey 


Feature  is  not  yet 
available,  but  can  be 
built  in  60  days. 

5 

'  Can  be  achieved 
through  creating  sub 
surveys  and  randomly 
directing  a  percent  of 
participants  to  each 
survey. 


Ability  to  have  different 
response  options  within  one 
matrix  question  (e.g., 
sometimes  N/A  and  other  times 
no  N/A) 


Ability  to  attach  external 
documents  to  the  dissemination 
email  (e.g.,  a  support  letter 
from  high  in  DoD)  or  ability  to 
imbed  email  signatures  into  the 
dissemination  email 


The  product  does 
not  allow  the 
attachment  of  external 
documents  to  the 
dissemination  email. 
However,  the  program 
is  able  to  include  a  link 
to  a  document  that  is 
posted  as  Web  page 
somewhere  else. 

”  Can  attach  any 
document  with  a  3  MB 
limit  per  file. 

§ 

'  Can  embed  a  link  in 
the  email  to  an  external 


Criteria 

# 


Criteria  Description 


Confirmit 

EFM 


Vovici  EFM 
Community 


WorldApp 
Key  Survey 


Zarca 

Interactive* 


Comments 


document. 


3.29  Ability  for  participant  to  clear  a 

X 

page  or  a  specific  question 
(e.g.,  radio  button) 

(3.5) 

(5) 

(NA) 

(5) 

3.30  Ability  to  export  data  directly 

^  x§ 

^  The  program  only 

to  SPSS  or  SAS 

(5) 

(5) 

(5) 

(4.5) 

allows  data  to  be 

exported  directly  to 
SPSS. 

^  For  SPSS,  there  is 
currently  the  need  to 
go  through  additional 
steps  to  export  to  a 
“.sav”  file  (i.e.,  SPSS 
data  file).  These  extra 
steps  will  be 
eliminated  in  the  next 
release  on  12/15/08. 
The  ability  to  export 
directly  SAS  is 
expected  in  January 
2009. 


System  automatically  creates  an 

x+ 

^  Plans  include  feature 

archive/back-up  of 
questionnaire/data 

(NA) 

(NA) 

(NA) 

(NA) 

to  be  added  in  2009. 

5 

'  Can  incorporate  a 
database  program  to 
create  a  real-time 
backup.  The  product 
saves  transaction  logs 
on  a  different  system. 

3.32  Advanced  Page  Layout  (e.g.,  •/  •/ 

resize  objects,  layer  objects,  (NA)  (NA)  (3)  (NA) 

snap  objects  to  a  grid,  and  lock 
a  page) 


3.33 

Ability  to  name  (label)  pages 

V 

X 

X 

(5) 

(NA) 

(NA) 

(NA) 

3.35 

Administrative  control,  such  as 

being  able  to  remove  cases  or 

(NA) 

(NA) 

(NA) 

(NA) 

reset  cases  (rather  than  going 
into  IMCEN  to  do  so)  to  allow 
participants  who  were 
inappropriately  exited  from  the 
survey  back  in 


^  This  is  performed 
through  HTML  only. 

5 

'  Can  do  this  through 
HTML,  but  not 
through  the  graphical 
user  interface  (GUI) 
platform. 

5 

'  Feature  is  not  yet 
available,  but  can  be 
built  in  60  days. 


Criteria 

# 

Criteria  Description 

Confirmit 

EFM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

Comments 

Percent  of  “Improves  Greatly” 
Features  Available 

71.88% 

62.50% 

71.88% 

50.00% 

“Improves  Greatly”  Feature  Average 
Usability  Rating 

4.31 

4.54 

4.22 

4.67 

“IMPROVES  SOMEWHAT”  FEATURES 

4.01 

Ability  to  add  graphics  in 

response  options 

(5) 

(4.5) 

(4) 

(2.5) 

4.03 

Option  of  exporting  only  list  of 

X 

X 

questions  VS  entire  survey 

(NA) 

(NA) 

(NA) 

(NA) 

Optical  scanning  features 

X 

xS 

§ 

*  Can  work  with  Zarca 

4.04 

(NA) 

(NA) 

(NA) 

(NA) 

in  order  to  get  optical 
scanning.  The  software 
allows  data  to  be 
imported  from  Excel. 

4.05 

Ability  for  platform  to 

X 

^  As  part  of  the  export 

automatically  dump  test  data 

(NA) 

(5) 

(NA) 

(NA) 

wizard,  the  feature 

once  activated 


gives  you  the  option  to 
export  all  data  (i.e.,  test 
and  real  data)  or  just 
real  data  (i.e.,  no  test 
data). 


4.06 

Feature  that  allow  developers  to 

V 

X 

V 

§ 

'  Within  the  preview. 

easily  check  if  skip  patterns 

(4.5) 

(NA) 

(NA) 

(NA) 

you  can  preview 

work  without  taking  entire 

specific  pages  with 

survey  as  participant 

functional  skips. 

4.07 

Ability  to  do  data  cleaning  and 

some  data  analysis  in  the 
application 

(NA) 

(NA) 

(NA) 

(NA) 

4.08 

Ability  to  display  basic 

figures/tables  (e.g.,  cross  tab)  of 
certain  important  questions 
(e.g.,  ability  to  look  at  non¬ 
responses  by  rank) 

(NA) 

(4) 

(NA) 

(NA) 

4.09 

Spell  check  library  where 

X 

5 

'  Feature  will  be 

commonly-used  words  or 

(NA) 

(5) 

(4) 

(NA) 

available  in  the  next 

acronyms  can  be  added 

release  on  12/15/08. 

4.10 

Meta-data  capabilities 

X 

(NA) 

(NA) 

(NA) 

(NA) 

Cri^ria  Criteria  Description 

Confirmit 

EFM 

Vovici  EFM 
Community 

WorldApp 
Key  Survey 

Zarca 

Interactive* 

Comments 

4.11  Ability  to  store  the  data  in  an 

X* 

*  Feature  is  not 

encrypted  format 

(NA) 

(NA) 

(NA) 

(NA) 

available,  but  can  built 
fairly  easily. 

4.12  Ability  to  create  professional- 

looking  emails 

(5) 

(4) 

(5) 

(4) 

4.13  Publish  a  survey  to  a 

X 

^  This  ability  can  be 

transportable  file  that  can  be 
run  on  local  machines  (e.g., 
non-Internet) 

(NA) 

(NA) 

(NA) 

(NA) 

performed  on  a 
portable  tablet,  but  not 
a  laptop. 

”  Feature  will  be  added 
in  next  release  in 
12/2008. 

4.14  Differentiates  in  tracking 

X 

X  " 

”  Plans  include  feature 

between  "partially  complete" 
and  "not  started"  and  filter 
emails  based  on  this 
differentiation 

(5) 

(NA) 

(NA) 

(NA) 

to  be  added  in  2009. 
s 

'  Feature  is  not 
available,  but  can  be 
built  in  30  days. 

Percent  of  “Improves  Somewhat” 
Features  Available 

84.62% 

69.23% 

84.62% 

61.54% 

“Improves  Somewhat”  Feature 

4.88 

4.50 

4.33 

3.25 

Average  Usability  Rating 

Total  Percent  of  Features  Available 

81.58% 

72.37% 

82.89% 

61.84% 

Total  Usability  Ratings  of  Features 

4.54 

4.64 

4.06 

4.28 

Feature  available;  Feature  not  available;  ♦♦♦  Feature  not  standard,  but  can  be  built;  Numbers  in  parentheses  indicate  average  usability 
ratings. 

*  Features  that  are  not  offered,  but  can  be  built  by  Zarca  will  not  cost  extra  as  long  as  they  are  indicated  in  the  initial  Statement  of  Work 
(SOW). 

Note;  Criteria  3.19,  3.23,  3.34,  3.36,  and  4.02  were  eliminated  because  of  duplication  with  other  criteria  listed. 
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Appendix  K 


Cost  Analysis 

Detailed  Cost  Estimates  for  COTS 
Survey  Software  Product  Finalists 


K-1 


Table  K-1 


Cost  of  Confirmit  EFM  Professional 


Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

BASE  FEATURES  COST 

Base  cost 

$22,500.00 

5 

Server  License 
includes  access  to 
Confirmit  Design, 
Deploy,  Export, 
Rapid  Results,  Data 
Processing,  etc.  for 
up  to  25 

Professional  users 

Developer  licenses 

$800.00 

5 

1  additional 
professional  user 

Reporting  licenses 

$0.00 

5 

Up  to  25 

Professional  users 

Reporting  licenses  are  not 
sold  separately  from 
developer  user  licenses. 

Cost  per  survey 

$0.50 

variable  (e.g., 
60,000) 

1  completed  survey 

Per  unit  price  reflects 
range  of  50,000  -  99,999 
units. 

Additional 

workgroups 

N/A 

N/A 

Upgrade  fees 

$0.00 

As  upgrades  are 
available 

Included  in  yearly  base 
cost. 

Installation  on  ARI 

servers 

$2,625.00  + 
travel  expenses 

1 

Yearly 

support/maintenance 

fees 

$0.00 

5 

Included  in  yearly  base 
cost. 

ADVANCED  FEATURES  COST 

API/XML  Web 

$13,500.00 

5 

services 


K-2 


Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

MobileSurvey  PC 
(offline  client  for 
laptops,  etc.) 

$180.00 

5 

1  seat 

Server  Security  (FTP 
&  Encryption) 

$4,500.00 

5 

Used  for  system 
integration  in  file  upload 
and  downloads. 

Image  File  Library 

$1,500.00 

5 

Recommended  to  aid  in 
including  any  graphics  in 
survey. 

PDF  Report  Exports 

$3,200.00 

5 

Used  to  export  reports  in 
PDF  format. 

Private  Labeling 
(removing  Vendor 
name  from  surveys 
and  emails) 

$0.00 

unlimited 

CUSTOM  SERVICES  COST 

Custom  program 
development 

$168.75 

variable 

1  hour 

TRAINING  COST 

Training  guides 

$0.00 

1 

2  full  sets  of 
documentation 

Additional  documentation 
can  be  purchased  at  $150 
per  set. 

Private  Training  (at 
ARI) 

$1,350.00 

1 

Up  to  6  students 

Non-Private  Training 

N/A 

N/A 

TOTAL  ESTIMATED  COST 

Estimated  Ist-year 
cost*: 

$58,075 

Estimated  5 -year 
cost*: 

$274,075 

Note:  Confirmit  is  on  the  GSA  schedule,  so  all  prices  reflect  GSA  unit  prices. 
Note:  Confirmit  is  willing  to  negotiate  price  by  a  small  margin  if  cost  is  an  issue. 


K-3 


*  Cost  for  Confirmit  EFM  Professional  includes  software  with  up  to  25  developer  and  reporting  licenses,  unlimited  number  of  surveys 
developed,  60,000  completed  survey  responses  per  year.  Image  File  Fibrary  add-on  capability,  a  Confirmit  representative's  assistance 
with  installation  on  ARI's  servers  (assuming  $1,000  in  travel  costs),  private  labeling,  and  2  days  of  private  training  for  up  to  6 
students. 
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Table  K-2 


Cost  of  Vovici  EFM  Community 


Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

BASE  FEATURES  COST 

Base  cost 

$9,495.00 

5 

1  workgroup  with  1 
administrator  user 
license 

A  workgroup  is  an 
independent  version  of 
the  platform  on  which  the 
users  develop  and 
administer  surveys. 

Developer  licenses 

$950.00 

5 

1  license 

Reporting  licenses 

$950.00 

5 

5  licenses 

Cost  per  survey 

$0.00 

Unlimited 

Additional 

workgroups 

$4,750.00 

5 

1  additional 
workgroup 

Upgrade  fees 

$0.00 

variable 

Installation  on  ARI 

servers 

$2,500.00  + 
travel  expenses 

1 

1  day  on  onsite 
assistance 

Remote  assistance  is 
included  in  base  cost. 

Yearly 

support/maintenance 

fees 

$0.00 

5 

unlimited 

Included  in  yearly  base 
cost. 

ADVANCED  FEATURES  COST 

API/XML  Web 
services 

$1,425.00 

5 

1  system 
synchronization 

MobileSurvey  PC 
(offline  client  for 
laptops,  etc.) 

$285.00 

5 

1  license 

Server  Security  (FTP 
&  Encryption) 

N/A 

N/A 

Image  File  Library 

$0.00 

5 

unlimited 

Size  and  amount  of 
pictures  depends  on  the 
server. 

PDF  Report  Exports 

$0.00 

5 

unlimited 

K-5 


Price 

Number  of 

j  Number 

times  charged  „ 

.  _  of  units 

in  5-year 

period 

Notes 

Private  Labeling 
(removing  Vendor 
name  from  surveys 
and  emails) 

$0.00 

unlimited 

CUSTOM  SERVICES  COST 

Custom  program 
development 

N/A 

N/A 

TRAINING  COST 

Training  guides 

$50.00 

1  1  guide 

Private  Training  (at 
ARI) 

$2,000.00  -f  1  1  day 

travel  expenses 

Non-Private  Training 

$1,320.35 

1  3-day  training;  per 

student 

In  Dulles,  VA. 

TOTAL  ESTIMATED  COST 

Estimated  Ist-year 
cost*: 

$18,995 

Estimated  5 -year 
cost*: 

$75,975 

Note:  Vovici  Corp  is  on  the  GSA  schedule,  so  all  prices  reflect  GSA  unit  prices. 

*  Cost  for  Vovici  EFM  Community  includes  1  workgroup  with  5  developer  licenses,  5  reporting  licenses,  unlimited  number  of 
surveys,  unlimited  number  of  responses,  a  Vovici  representative's  assistance  with  installation  on  ARl's  servers  (with  estimated  $100  of 
Vovici  representative  travel  costs),  1  paper  training  guide,  and  1  day  of  private  training  (with  estimated  $100  of  Vovici  representative 
travel  costs). 
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Table  K-3 


Cost  of  WorldApp  Key  Survey 


Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

BASE  FEATURES  COST 

Base  cost 

$20,000.00 

1 

1  administrative 
user  license 

Developer  licenses 

$4,200.00 

1 

1  license 

Reporting  licenses 

Cost  per  survey 

$0.00 

$0.00 

1 

unlimited 

Reporting  licenses  are  not 
sold  separately  from 
developer  user  licenses. 

Additional 

workgroups 

N/A 

N/A 

Upgrade  fees 

$0.00 

variable 

Installation  on  ARI 

servers 

$150.00 

1 

1  hour  onsite 
assistance 

Installation  should  take 
about  1-2  days. 

Yearly 

support/maintenance 

fees 

18%  of  1st  year 
product  and 
features  cost 

5 

ADVANCED  FEATURES  COST 

API/XML  Web 
services 

$7,000.00 

1 

MobileSurvey  PC 
(offline  cilent  for 
laptops,  etc.) 

N/A 

N/A 

Server  Security  (FTP 
&  Encryption) 

N/A 

N/A 

Image  File  Library 

$0.00 

1 

unlimited 

PDF  Report  Exports 

$0.00 

1 

unlimited 

Private  Labeling 

$2,000.00 

1 

unlimited 

(removing  Vendor 
name  from  surveys 
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Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

and  emails) 

CUSTOM  SERVICES  COST 

Custom  program 
development 

$200.00 

variable 

1  hour 

TRAINING  COST 

Training  guides 

$50.00 

1 

1  guide 

Private  Training  (at 
ARI) 

$1,500.00 

1 

1  day 

Non-Private  Training 

$150.00 

1  hour 

Up  to  25  students 

Web-based  training;  2-4 
hours  suggested. 

_ TOTAL  ESTIMATED  COST 

Estimated  Ist-year  $49,029 
cost*: 


Estimated  5-year  $78,945 

cost*: 


*  Cost  for  WorldApp  Key  Survey  includes  software  with  5  developer  and  reporting  licenses,  unlimited  number  of  surveys,  unlimited 
number  of  responses,  a  WorldApp  representative's  assistance  with  installation  on  ARl's  servers  (8  hours),  private  labeling,  1  paper 
training  guide,  and  1  day  of  private  training  for  up  to  25  students. 
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Table  K-4 


Cost  of  Zarca  Interactive 


Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

BASE  FEATURES  COST 

Base  cost 

$65,000.00 

1 

1  software  with 
unlimited  users. 

Developer  licenses 

$0.00 

1 

unlimited 

Reporting  licenses 

$0.00 

1 

unlimited 

Reporting  licenses  are  not 
sold  separately  from 
developer  user  licenses. 

Cost  per  survey 

$0.00 

unlimited 

Additional 

workgroups 

N/A 

N/A 

Upgrade  fees 

$0.00 

variable 

Installation  on  ARI 

servers 

$15,000.00 

1 

Yearly 

support/maintenance 

fees 

$15,000.00 

5 

ADVANCED  FEATURES  COST 

API/XML  Web 
services 

$5,000.00 

Varies  depending  on 

ARI's  servers  and  how 
much  programming  is 
needed. 

MobileSurvey  PC 
(offline  client  for 
laptops,  etc.) 

N/A 

N/A 

Server  Security  (FTP 
&  Encryption) 

N/A 

N/A 

Image  File  Library 

N/A 

N/A 

PDF  Report  Exports 

N/A 

N/A 
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Price 

Number  of 
times  charged 
in  5-year 
period 

Number 
of  units 

Notes 

Private  Labeling 
(removing  Vendor 
name  from  surveys 
and  emails) 

$7,000.00 

1 

unlimited 

CUSTOM  SERVICES  COST 

Custom  program 
development 

$135.00 

variable 

1  hour 

Only  applies  to 
customization  after  SOW 
is  written;  Any 
customization  written  into 
the  initial  SOW  is 
included  in  base  charge. 

TRAINING  COST 

Training  guides 

$0.00 

Guide  is  available 
electronically. 

Private  Training  (at 
ARI) 

$0.00 

1 

1  day 

Initial  training. 

Non-Private  Training 

$0.00 

5  Unlimited  number 

of  students 

Web-based  training; 
Available  for  training 
users  after  initial  training 
is  given. 

TOTAL  ESTIMATED  COST 

Estimated  Ist-year 
cost*: 

$102,000 

Estimated  5 -year 
cost*: 

$162,000 

*  Cost  for  Zarca  includes  software  with  unlimited  developer  and  reporting  licenses,  unlimited  number  of  surveys,  unlimited  number  of 
responses,  a  Zarca  representative's  assistance  with  installation  on  ARI's  servers,  private  labeling,  and  all  training  (including  electronic 
training  guides,  initial  private  training  at  ARI,  and  follow-up  Web-based  training  for  users  later  added. 
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